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Executive  Summary 


Purpose 


In  the  early  1990s,  the  National  Aeronautics  and  Space  Administration 
(NASA)  administrator  challenged  the  agency  to  complete  projects  faster, 
better,  and  cheaper.  The  intent  was  to  reduce  costs,  become  more 
efficient,  and  increase  scientific  results  by  conducting  more  and  smaller 
missions  in  less  time.  Although  NASA  maintained  a  high  success  rate 
under  the  faster,  better,  and  cheaper  strategy,  a  few  significant  mission 
failures  also  occurred — particularly  the  loss  of  the  Mars  Polar  Lander  and 
Climate  Orbiter  spacecraft.  NASA  investigations  of  these  failures,  as  well 
as  its  review  of  other  programs,  raised  concern  that  lessons  from  past 
experiences  were  not  being  applied  to  current  programs  and  projects. 

At  the  request  of  the  Chairman  and  Ranking  Minority  Member, 
Subcommittee  on  Space  and  Aeronautics,  House  Committee  on  Science, 
GAO  assessed  whether  NASA  has  adequate  mechanisms  in  place  to  ensure 
that  past  lessons  learned  from  mission  failures  are  being  applied. 
Specifically,  GAO  (1)  identified  the  policies,  procedures,  and  systems 
NASA  has  in  place  for  lessons  learning,  (2)  assessed  how  effectively  these 
policies,  procedures,  and  systems  facilitate  lessons  learning,  and  (3) 
determined  whether  further  efforts  are  needed  to  improve  lessons 
learning. 


Background 


NASA’s  procedures  and  guidelines  require  that  program  and  project 
managers  review  and  apply  lessons  learned  from  the  past  throughout  a 
program’s  or  project’s  life  cycle  and  to  document  and  submit  any 
significant  lessons  to  the  agency’s  Lessons  Learned  Information  System 
(LLIS)  in  a  timely  manner.  NASA  defines  a  lesson  learned  as  knowledge  or 
understanding  gained  by  experience.  The  experience  may  be  positive,  such 
as  a  successful  test  or  mission,  or  negative,  such  as  a  mishap  or  failure.  A 
lesson  must  be  significant  in  that  it  has  a  real  or  assumed  impact  on 
operations;  valid  in  that  it  is  factually  correct;  and  applicable  in  that  it 
identifies  a  specific  design,  process,  or  decision  that  reduces  or  eliminates 
the  potential  for  failures  and  mishaps,  or  reinforces  a  positive  result. 


Results  in  Brief 


NASA  recognizes  the  importance  of  learning  from  the  past  to  ensure  future 
mission  success  and  uses  several  mechanisms  to  capture  and  disseminate 
lessons  learned.  The  principal  source  NASA  has  established  for  the 
agency-wide  collection  and  sharing  of  lessons  is  the  LLIS,  a  Web-based 
lessons  database  that  managers  are  required  to  review  on  an  ongoing 
basis.  In  addition,  NASA  uses  training,  program  reviews,  and  periodic 
revisions  to  agency  policies  and  guidelines  to  communicate  lessons. 
Several  NASA  centers  and  key  programs  also  maintain  lessons  learned 
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systems  that  are  geared  toward  their  own  staff.  Recently,  NASA  has  taken 
steps  to  improve  the  way  it  captures  and  shares  information  by  developing 
a  business  strategy  called  knowledge  management.  Knowledge 
management  can  be  defined  as  the  way  that  organizations  create,  capture, 
and  reuse  knowledge  to  achieve  their  objectives.  According  to  NASA 
officials,  knowledge  management  has  the  potential  to  link  agency  staff 
with  the  knowledge  and  resources  they  need  to  complete  tasks  faster, 
better,  and  cheaper.  In  pursuit  of  knowledge  management,  NASA  has 
developed  a  strategic  plan,  established  a  management  team  to  coordinate 
knowledge  management  activities  at  NASA’s  centers,  and  initiated  several 
information  technology  pilot  projects. 

Despite  the  processes  and  procedures  in  place  to  capture  and  share 
lessons  learned,  there  is  no  assurance  that  lessons  are  being  applied 
toward  future  missions  success.  A  GAO  survey  of  NASA  program  and 
project  managers  revealed  weaknesses  in  the  collection  and  sharing  of 
lessons  learned  agency-wide.  While  some  lessons  learning  does  take  place, 
our  survey  found  that  lessons  are  not  routinely  identified,  collected,  or 
shared  by  programs  and  project  managers.  Respondents  reported  that  they 
are  unfamiliar  with  lessons  generated  by  other  centers  and  programs.  In 
addition,  many  respondents  indicated  that  they  are  dissatisfied  with 
NASA’s  lessons  learned  processes  and  systems.  Managers  also  identified 
challenges  or  cultural  barriers  to  the  sharing  of  lessons  learned,  such  as 
the  lack  of  time  to  capture  or  submit  lessons  and  a  perception  of 
intolerance  for  mistakes.  They  further  offered  suggestions  for  areas  of 
improvement,  including  enhancements  to  LLIS  and  implementing 
mentoring  and  “storytelling,”  or  after-action  reviews,  as  additional 
mechanisms  for  lessons  learning. 

While  NASA’s  current  knowledge  management  efforts  should  lead  to  some 
improvement  in  the  sharing  of  agency  lessons  and  knowledge,  they  lack 
ingredients  that  have  been  shown  to  be  critical  to  the  success  of 
knowledge  management  at  leading  organizations.  Cultural  resistance  to 
sharing  knowledge  and  the  lack  of  strong  support  from  agency  leaders 
often  make  it  difficult  to  implement  an  effective  lessons  learning  and 
knowledge  sharing  environment.  We  found  that  successful  industry  and 
government  organizations  have  overcome  barriers  by  making  a  strong 
management  commitment  to  knowledge  sharing,  developing  a  well- 
defined  business  plan  for  implementing  knowledge  management, 
providing  incentives  to  encourage  knowledge  sharing,  and  building 
technology  systems  to  facilitate  easier  access  to  information.  The 
application  of  these  principles  could  increase  opportunities  for  NASA  to 
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perform  its  basic  mission  of  exploring  space  faster,  better,  and  cheaper 
more  successfully. 


Principal  Findings 


NASA’s  Policies  and 
Procedures  for  Lessons 
Learning 


NASA  uses  various  mechanisms  to  communicate  lessons  garnered  from 
past  programs  and  projects.  Policies  and  guidelines,  programmatic  and 
technical  reviews,  mentoring  and  training  programs,  the  Academy  of 
Program  and  Project  Leadership,  and  LLIS  are  the  mechanisms  employed 
by  NASA  for  capturing  and  sharing  lessons  learned.  LLIS  is  the  “official” 
agency-wide  repository  for  such  lessons.  Lessons  entered  in  the  LLIS 
database  are  screened  for  relevance  and  to  ensure  that  they  do  not  contain 
sensitive  or  proprietary  information.  Initial  reviews  of  lessons  are  usually 
conducted  by  the  centers,^  with  a  final  review  by  the  Office  of  Safety  and 
Mission  Assurance.  After  a  lesson  is  entered  into  the  system,  it  remains  in 
the  database  indefinitely  and  is  not  reviewed  for  currency  or  relevance. 
Currently,  the  system  contains  over  900  lessons  on  topics  ranging  from 
program  management  to  technical  cause  of  failure. 

In  response  to  the  Mars  Program  failures  and  the  recommendations  of 
agency  reviews  of  program  and  project  execution,  NASA  has  recently 
taken  action  to  improve  its  policies  and  practices  for  capturing  and 
sharing  knowledge  by  developing  a  business  strategy  referred  to  as 
knowledge  management.  Implementation  of  knowledge  management  can 
lead  to  increased  productivity,  collaboration,  and  innovation  in  the 
workplace.  To  coordinate  and  guide  its  efforts,  NASA  recently  formed  a 
knowledge  management  team,  which  developed  a  strategic  plan  that  laid 
out  broad  goals  and  objectives  for  knowledge  management.  In  addition, 
several  pilot  projects  are  underway  at  various  NASA  centers  to  enhance 
knowledge  sharing. 


^  NASA  consists  of  NASA  headquarters,  nine  centers,  the  Jet  Propulsion  Laboratory 
(operated  under  contract  to  NASA  by  the  California  Institute  of  Technology),  and  several 
ancillary  installations  and  offices  in  the  United  States  and  abroad.  The  implementation  of 
NASA  programs  and  aeronautical  and  space/earth  science  research  occurs  primarily  at  the 
centers. 
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Fundamental  Weaknesses 
Exist  in  the  Collection  and 
Sharing  of  Lessons 
Learned 


A  survey  we  conducted  of  all  NASA  program  and  project  managers 
revealed  fundamental  weaknesses  in  the  collection  and  sharing  of  lessons 
learned  agency-wide.  Although  NASA’s  processes  and  procedures  require 
that  program  and  project  managers  review  and  apply  lessons  learned 
throughout  a  program’s  or  project’s  life  cycle,  our  survey  found  that 
managers  do  not  routinely  identify,  collect,  or  share  lessons.  Respondents 
indicated  that  LLIS,  NASA’s  primary  method  for  disseminating  lessons 
learned  agency-wide,  is  not  the  primary  source  for  lessons  learning. 
Instead,  managers  identified  program  reviews  and  informal  discussions 
with  colleagues  as  their  principal  sources  for  lessons  learned.  One  reason 
LLIS  is  not  widely  used,  according  to  one  center  official,  is  because  its 
lessons  cover  so  many  topics  that  it  is  difficult  to  search  for  an  applicable 
lesson.  Another  respondent  indicated  that  it  is  difficult  to  weed  through  all 
the  irrelevant  lessons  to  get  to  the  few  “jewels”  that  you  need  to  find. 


Respondents  also  identified  challenges  or  cultural  barriers  to  the  sharing 
of  lessons  learned  as  well  as  areas  of  improvement.  Managers  noted  that 
there  is  a  reluctance  to  share  negative  lessons  for  fear  that  they  might  not 
be  viewed  as  good  project  managers,  and  there  is  a  lack  of  time  for  lessons 
learning  to  take  place.  One  manager  stated,  “Until  we  can  adopt  a  culture 
that  admits  frankly  to  what  really  worked  and  didn’t  work,  1  find  many  of 
these  tools  to  be  suspect.”  Managers  suggested  that  NASA  could  improve 
lessons  learning  by  implementing  mentoring  and  “storytelling”  activities, 
and  it  could  enhance  LLIS  by  increasing  its  search  functions,  including 
more  positive  lessons,  and  developing  a  mechanism  to  disseminate  key 
lessons  to  users. 


In  discussions  with  NASA  officials,  we  found  there  was  general  agreement 
with  the  results  of  our  survey  as  well  as  suggested  improvements  for 
lessons  learning.  Officials  indicated  that  lessons  learning  has  taken  on 
greater  importance  in  recent  years  due  to  the  implementation  of  more 
programs  and  projects  under  the  faster,  better,  cheaper  strategy  and  the 
continuing  loss  of  agency  expertise  due  to  attrition.  They  acknowledge 
that  LLIS  has  not  been  an  effective  mechanism  for  agency-wide  sharing  of 
lessons.  Although  the  system  is  viewed  as  providing  a  useful  repository  for 
storing  lessons,  officials  agreed  with  managers’  concerns  about  the 
difficulties  involved  in  searching  the  system  and  finding  relevant  lessons, 
the  inconsistent  quality  of  information  contained  in  the  system,  and  the 
lack  of  lessons  about  positive  project  experiences.  However,  while 
program  and  project  managers’  suggested  improvements  would  help 
increase  the  usability  of  LLIS,  they  have  not  targeted  some  of  the  more 
fundamental  problems  hampering  NASA’s  ability  to  share  lessons,  such  as 
persistent  cultural  barriers. 
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Creating  an  Environment 
for  Lessons  Learning 
through  Knowledge 
Management 


Leading  organizations  are  discovering  that  actively  managing  knowledge 
creates  value  by  increasing  productivity  and  fostering  innovation. 

Likewise,  NASA’s  paramount  concern  should  be  about  capturing  and 
sharing  organizational  knowledge  and  using  it  to  perform  its  basic  mission 
of  exploring  space  faster,  better,  and  cheaper.  Although  NASA  has  recently 
taken  action  to  improve  the  way  in  which  the  agency  captures,  organizes, 
and  shares  knowledge,  these  efforts  do  not  fully  address  the  fundamental 
weaknesses  in  lessons  learning  identified  by  our  survey:  namely,  cultural 
resistance  to  sharing  knowledge  and  the  lack  of  an  effective  strategic 
framework  and  management  attention  for  overcoming  such  resistance. 


NASA  has  made  a  reasonable  start  by  developing  a  strategic  plan  for 
knowledge  management,  but  the  agency  has  not  made  a  good  business 
case  for  how  it  will  implement  and  use  knowledge  management  within  the 
organization.  In  addition,  while  successful  industry  and  government 
organizations  have  made  a  firm  commitment  to  making  knowledge 
management  practices  work,  NASA  has  not  provided  the  leadership, 
support,  and  resources  needed  for  effective  knowledge  management  to 
take  place.  Furthermore,  knowledge  management  organizations  have 
employed  incentives,  processes,  and  systems  designed  to  address  cultural 
barriers  to  continuous  lessons  learning  and  knowledge  sharing.  For 
example,  organizations  that  value  knowledge  sharing  have  encouraged 
employees  to  spend  time  sharing  knowledge,  helped  facilitate 
communities  of  practice  based  around  common  interests,  and  provided 
rewards  when  knowledge  has  been  shared  and  applied.  NASA  has  not 
done  so  on  an  agency-wide  basis. 


Recommendations 


NASA  needs  to  strengthen  its  lessons  learning  in  the  context  of  its  overall 
efforts  to  develop  and  implement  an  effective  knowledge  management 
program.  Improvement  of  NASA’s  lessons  learning  processes  and  systems 
can  help  to  ensure  that  knowledge  is  gained  from  past  experiences  and 
applied  to  future  missions. 

We  recommend  that  the  NASA  administrator  strengthen  the  agency’s 
lessons  learning  processes  and  systems  by 

articulating  the  relationship  between  lessons  learning  and  knowledge 
management  through  an  implementation  plan  for  knowledge  management; 

designating  a  lessons  learned  manager  to  lead  and  coordinate  all  agency 
lessons  learning  efforts; 
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•  establishing  functional  and  technical  linkages  among  the  various  center- 
level  and  program-level  lessons  learning  systems; 

•  developing  ways  to  broaden  and  implement  mentoring  and  “storytelling” 
as  additional  mechanisms  for  lessons  learning; 

•  identifying  incentives  to  encourage  more  collection  and  sharing  of  lessons 
among  employees  and  teams,  such  as  links  to  performance  evaluations 
and  awards; 

•  enhancing  LLIS  by  coding  information  and  developing  an  easier  search 
capability  to  allow  users  to  identify  relevant  lessons,  including  more 
positive  lessons,  providing  a  means  to  disseminate  key  lessons  to  users; 
and  soliciting  user  input  on  an  ongoing  basis;  and 

•  tracking  and  reporting  on  the  effectiveness  of  the  agency’s  lessons 
learning  efforts  using  objective  performance  metrics. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  NASA  generally  concurred 
with  our  recommendations  for  improving  the  agency’s  lessons  learned 
processes  and  systems.  NASA  stated  that  it  must  do  a  better  job  of 
communicating  the  various  lessons  learned  sources  to  employees, 
improving  mechanisms  to  link  these  sources,  and  ensuring  appropriate 
training  for  employees  in  order  to  maximize  lessons  learning. 
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Chapter  1:  Introduction 


The  1990’s  was  a  decade  of  significant  challenge  for  the  National 
Aeronautics  and  Space  Administration  (NASA)  brought  about,  in  part,  by 
decreased  budgets,  a  reduced  work  force,  and  technological  innovation. 

To  adapt  to  this  changing  environment,  the  NASA  administrator 
challenged  agency  personnel  to  do  projects  faster,  better,  and  cheaper 
(FBC)  by  streamlining  practices  and  becoming  more  efficient.  The  goal 
was  to  shorten  program  development  times,  reduce  cost,  and  increase 
scientific  return  by  flying  more  and  smaller  missions  in  less  time. 

However,  the  failure  of  the  Mars  Polar  Lander  and  Climate  Orbiter 
spacecraft  raised  concerns  that  lessons  learned  from  past  experiences 
were  not  being  passed  along  and  applied  toward  future  mission  success. 

To  ensure  that  individuals  learn  from  past  experiences,  effective  processes 
and  systems  must  be  in  place  to  collect,  store,  and  disseminate  lessons 
learned. 


Faster,  Better, 
Cheaper:  A 
Management 
Philosophy 


In  1992  NASA  adopted  the  FBC  philosophy  as  a  way  of  managing 
programs  and  projects.  An  important  element  of  this  approach  was  a 
reduction  in  NASA  headquarters  management  and  moving  more  program 
responsibility  to  NASA’s  centers.  This  philosophy  also  increased  the 
demand  for  program  and  project  managers  at  a  time  when  NASA  was 
experiencing  a  significant  reduction  in  staff  due  to  retirements, 
downsizing,  and  departures  to  industry.  Prior  to  FBC,  there  were  fewer 
missions,  and  program  and  project  managers  accumulated  significant  first¬ 
hand  experience  before  managing  a  program.  Under  FBC,  with  a  threefold 
increase  in  projects  and  fewer  staff,  this  was  not  always  the  case. 
Relatively  unseasoned  managers  who  were  challenged  to  be  more  efficient 
and  innovative  and  to  take  greater  risks  in  designing  and  implementing 
missions  led  many  projects. 

NASA’s  record  in  designing,  developing,  and  operating  smaller  spacecraft 
under  the  FBC  concept,  while  mostly  successful,  has  experienced  a  few 
notable  failures.  A  number  of  projects  including  Clementine,  Near  Asteroid 
Rendezvous,  Mars  Pathfinders,  and  Mars  Global  Surveyor  proved 
successful  within  the  schedule  and  fiscal  constraints  imposed  under  the 
FBC  concept.  However,  for  other  programs,  including  the  Mars  Polar 
Lander  and  Climate  Orbiter,  the  challenge  was  too  great,  resulting  in 
undue  risk-taking  and  failure.  After-action  reports  commissioned  by  NASA 
found  that  the  Mars  Program  failures  resulted  from  cost  and  schedule 
constraints  and  a  lack  of  rigorous  attention  to  sound  process  and 
practices. 
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Mars  Surveyor 
Program 


In  1993  NASA  initiated  the  Mars  Surveyor  Program  with  the  objective  of 
conducting  an  on-going  series  of  missions  to  explore  Mars.  NASA’s  Jet 
Propulsion  Laboratory  (JPL)  was  selected  as  the  lead  center  for  this 
program  and  was  responsible  for  mission  design,  spacecraft  and  payload 
development,  as  well  as  integration,  testing,  and  launch  of  the  proposed 
spacecraft. 

The  first  Mars  Surveyor  Program  spacecraft,  the  Mars  Global  Surveyor  and 
the  Mars  Pathfinder,  were  launched  in  1996  and  both  were  highly 
successful.  After  completing  its  primary  mission  of  acquiring  knowledge  of 
the  climate,  subsurface  resources,  and  topography  of  Mars,  the  Mars 
Global  Surveyor  is  conducting  reconnaissance  of  future  Mars  landing  sites. 
The  Mars  Pathfinder  and  Sojourner  Rover  were  a  demonstration  of  a  way 
to  use  technology  to  deliver  an  instrumented  lander  and  a  free-ranging 
robotic  rover  to  the  surface  of  Mars.  Two  additional  Mars  Surveyor 
Program  spacecraft  were  launched  in  late  1998  and  early  1999, 
respectively,  to  explore  and  collect  additional  scientific  data  on  Mars. 

The  Mars  Climate  Orbiter,  which  cost  $75  million  to  develop,  was 
launched  December  11,  1998,  and  was  intended  to  observe  Mars’  seasonal 
climate  and  daily  weather  from  a  low  orbit  around  the  planet.  Nine  and  a 
half  months  after  launch,  in  September  1999,  the  spacecraft  was  supposed 
to  fire  its  main  engine  to  achieve  an  elliptical  orbit  around  Mars  and  then 
pass  through  its  upper  atmosphere  for  several  weeks  before  moving  into  a 
low  circular  orbit  around  the  planet.  On  September  23,  1999,  the  Orbiter 
spacecraft  was  destroyed  when  it  entered  the  Martian  atmosphere  on  a 
lower  than  expected  trajectory,  causing  the  spacecraft  to  bum  up.  A  NASA 
mishap  investigation  team  determined  that  the  loss  was  due  to  the 
mistaken  use  of  English  rather  than  metric  units  in  the  navigation 
software.  As  a  result,  an  incorrect  trajectory  was  computed  and  the 
spacecraft  was  lost. 

The  Mars  Polar  Lander  was  launched  January  3,  1999,  and  cost  $113 
million  to  develop.  The  Lander  was  a  robotic  spacecraft  designed  to  land 
softly  near  the  South  Pole  of  Mars  to  study  the  planet’s  layered  polar 
terrain.  Attached  to  the  Lander  was  the  Deep  Space  2  mission — two  small 
probes  designed  to  separate  from  the  Lander  about  5  minutes  prior  to 
touchdown  and  penetrate  at  high  speed  into  the  Martian  soil  to  search  for 
traces  of  vaporized  water  ice.  The  Deep  Space  2  probes  cost  $29.6  million 
to  develop.  Unfortimately,  the  Lander  was  lost  December  3,  1999,  while 
attempting  to  land.  The  two  small  probes  also  were  lost.  NASA’s  mishap 
investigation  team  concluded  that  the  premature  shutdown  of  the  Lander’s 
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descent  engines  was  the  most  probable  cause  for  the  loss  of  the 
spacecraft. 


Mishap  Investigations 
Raised  Concern  That 
NASA  Did  Not  Heed 
Lessons  Learned 


The  failure  of  the  Mars  Polar  Lander  and  Climate  Orbiter  spacecraft  raised 
concern  that  lessons  learned  from  past  mishaps  and  programs  were  not 
being  applied  effectively  toward  future  mission  success.  Support  for  this 
comes  from  reports  commissioned  by  NASA  that  reviewed  spacecraft 
failures  in  the  Mars  Program,  Shuttle  wiring  problems,  and  an  assessment 
of  the  agency’s  approach  in  executing  “Faster,  Better,  Cheaper”  projects.^ 
These  reports  identified  the  root  and  contributing  causes  for  failure  as 
well  as  making  broader  recommendations  on  ways  NASA  might  improve 
its  general  approach  to  executing  programs  and  projects. 


NASA’s  decision  to  leave  critical  information-gathering  mechanisms  off 
the  Mars  Polar  Lander  is  a  case  in  point.  The  importance  of  downlink 
telemetry^  during  critical  mission  events  was  a  lesson  learned  from  the 
failure  of  NASA’s  1992  Mars  Observer  mission  that  was  not  heeded  during 
design  of  the  Mars  Polar  Lander  seven  years  later.  The  design  of  Lander 
precluded  transmission  of  critical  communications  or  telemetry  data 
during  entry,  descent,  and  landing  on  Mars.  A  Special  Review  Board 
determined  that  the  probable  cause  of  the  loss  of  the  spacecraft  was  due 
to  a  premature  shutdown  of  the  descent  engines  due  to  spurious  signals 
generated  when  the  Lander’s  legs  deployed.  The  spurious  signals  gave  a 
false  indication  that  the  spacecraft  had  landed.  The  board  also  determined 
that  the  spurious  signals  resulted  from  inadequate  software  design  and 
systems  testing. 


Concern  that  NASA  had  not  heeded  past  lessons  learned  was  also 
confirmed  in  a  recent  study^  conducted  by  RAND’s  National  Security 


^  Mars  Program  Independent  Assessment,  chaired  by  Mr.  A.  Thomas  Young  (retired), 
Lockheed  Martin;  Mars  Climate  Orbiter  (MCO)  Mishap  Investigation,  chaired  by  Mr.  Arthur 
Stephenson,  director,  Marshall  Space  Flight  Center;  NASA  Feister,  Better,  Cheaper  Task 
Force,  chaired  by  Mr.  Anthony  Spear  (retired).  Jet  Propulsion  Laboratory;  and  Shuttle 
Independent  Assessment,  chaired  by  Dr.  Henry  McDonald,  director,  Ames  Research  Center. 

^  Telemetry  is  information  on  the  condition  of  the  spacecraft  and  its  subsystems,  such  as 
the  temperature  and  voltage  of  spacecraft  batteries,  that  is  transmitted  to  spacecraft 
operators  on  the  ground. 

^  Sarsfield,  Liam.  The  Application  of  Best  Practices  to  Unmanned  Spacecraft  Development: 
An  Exploration  of  Success  and  Failure  in  Recent  Missions.  (Santa  Monica,  CA:  RAND, 
2000). 
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Research  Division.  The  purpose  of  the  study  was  to  provide  guidance  on 
practices  that  reduce  risk  and  improve  the  performance  of  next-generation 
spacecraft  by  including  an  examination  of  NASA’s  successes  and  failures 
in  building  spacecraft  both  before  and  after  implementing  the  faster, 
better,  cheaper  approach.  The  RAND  study  identified  the  top  ten  sources 
of  failures  in  NASA  programs  and  reported  that  a  significant  source  of 
error  has  been  the  failure  of  NASA  to  incorporate  lessons  previously 
learned  and  consistently  apply  them.  As  shown  in  the  following  table,  the 
reasons  for  failure  of  the  Mars  Polar  Lander  and  Climate  Orbiter  are  not 
new;  they  have  been  identified  as  areas  of  concern  by  major  program 
reviews  and  have  occurred  during  the  development  of  several  other  space 
systems. 
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Table  1 :  Reasons  for  Spacecraft  Failures 

Reasons  for  Failure 

Major  Program  Reviews 

Major  Mishap  Reviews 

Broad 

Area 

Review 

Lockheed 

Martin 

Independent 

Assessment 

Team 

Faster, 

Better, 

Cheaper 

Task 

Force 

W  ^  Mars 

I  Climate 

R  Orbiter 

E 

Mars 

Polar 

Lander 

Lewis 

O 

H 

O 

Mars 

Observer 

DC-X® 

Challenger 

Cost  and  Schedule 
Constraints 

• 

• 

• 

• 

• 

Insufficient  Risk 
Assessment  and 
Planning 

• 

• 

• 

• 

• 

• 

• 

• 

• 

Underestimation 
of  Complexity  and 
Technology 

Maturity 

• 

• 

• 

• 

• 

Insufficient 

Testing 

• 

•  • 

• 

• 

• 

• 

Poor  Team 
Communication 

• 

• 

• 

•  • 

• 

• 

• 

• 

Inattention  to 
Quality  and  Safety 

• 

• 

• 

• 

• 

Inadequate 

Review 

Process 

• 

• 

• 

•  • 

• 

• 

• 

• 

• 

Design  Errors 

• 

•  • 

• 

• 

• 

• 

Inadequate 

System 

Engineering 

• 

• 

• 

•  • 

• 

• 

• 

• 

Inadequate  or 

Under  Trained 

Staff 

• 

• 

• 

• 

• 

• 

• 

• 

“WIRE:  Wide-Field  Infrared  Explorer 
“SOHO:  Solar  and  Heliospheric  Observatory 
“DC-X:  Delta  Clipper-Experimental 


Source:  RAND,  used  with  permission. 
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NASA  Efforts  to 
Apply  Lessons 
Learned  from  Mars 
Mishaps 


In  December  2000,  NASA  issued  a  report,  Enhancing  Mission  Success, 
which  identified  specific  actions  the  agency  planned  to  take  to  apply  prior 
mishap  findings  and  recommendations  broadly  in  order  to  improve  its 
approach  to  executing  programs  and  projects/  The  development  of  this 
report  was  driven  principally  by  the  multiple  mission  failures  associated 
with  the  Mars  Program.  As  part  of  its  assessment,  NASA  reviewed  a  total 
of  165  recommendations  originating  from  the  Mars  mishaps  reports,  the 
Shuttle  Independent  Assessment,  and  the  agency-wide  assessment  of 
faster,  better,  cheaper  projects.  As  a  result,  NASA  recommended  17 
actions  that  are  integrated  into  five  broad  themes:  people,  technology,  risk 
management,  program  and  project  management,  and  communications. 

One  of  the  17  recommended  actions,  which  deals  with  improving 
communications,  concerns  knowledge  management  and  the  recognition 
that  NASA  needs  to  do  better  in  capturing,  disseminating,  and  utilizing 
knowledge.  This  includes  improving  the  capture  and  application  of  lessons 
learned  from  programs,  projects,  and  missions,  with  the  goal  of  ensuring 
that  NASA  does  not  have  to  keep  “relearning”  the  lessons  of  the  past¬ 
relearning  evidenced  by  the  reoccurrence  of  similar  causes  to  mission 
failures  or  difficulties.  The  report  also  indicated  a  lack  of  access  to  and 
process  for  lessons  learned.  The  recommended  action  further  stated  that 
the  continuous  capture  and  application  of  project  knowledge  and  lessons 
learned  must  become  a  core  business  process  within  the  agency’s  program 
and  project  management  environment. 


Lessons  Learning:  A 
Mechanism  to  Learn 
from  Successes  As 
Well  As  Mistakes 


Use  of  lessons  learned  is  a  principal  component  of  an  organizational 
culture  committed  to  continuous  improvement.  Lessons  learned 
mechanisms  serve  to  communicate  acquired  knowledge  more  effectively 
and  to  ensure  that  beneficial  information  is  factored  into  planning,  work 
processes,  and  activities.  Lessons  learned  provide  a  powerful  method  of 
sharing  good  ideas  for  improving  work  processes,  facility  or  equipment 
design  and  operation,  quality,  safety,  and  cost-effectiveness. 


National  Aeronautics  and  Space  Administration.  Enhancing  Mission  Success  -  A 
Framework  for  the  Future.  A  Report  by  the  NASA  Chief  Engineer  and  the  NASA  Integrated 
Action  Team,  December  21,  2000. 

^  Knowledge  management  can  be  defined  as  the  way  that  organizations  create,  capture,  and 
re-use  knowledge  to  achieve  organizational  objectives. 
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Elements  of  the  Lessons  The  mechanisms  or  processes  used  to  collect,  share,  and  disseminate 

Learned  Process  lessons  learned  may  vary,  but  in  general  such  a  process  is  comprised  of 

four  main  elements:  collection,  verification,  storage,  and  dissemination. 
Figure  1  is  a  generic  representation  of  the  lessons  learned  process. 


Figure  1:  Generic  Lessons  Learned  Process 


Source:  Based  on  Weber,  R.,  Aha,  D.,  and  Becerra-Fernanadez,  I.  Categorizing  inteiiigent  Lessons 
Learned  Systems.  Intelligent  Lessons  Learned  Systems:  Papers  from  the  AAAI  Workshop  (Technicai 
Report  AiC-00-005).  Aha,  D.W.  and  Weber,  R.  (Eds.)  pp.  63-67.  Washington,  DC:  Navai  Research 
Laboratory,  Navy  Center  for  Appiied  Research  in  Artificiai  inteiiigence,  2000. 


•  The  collection  process  involves  the  capture  of  information  through 
structured  and  unstructured  processes  such  as  mishap  or  accident 
reporting,  project  critiques,  written  forms,  and  meetings.  The  collection 
of  lessons  may  come  from  as  many  sources  as  an  organization  is  willing 
to  solicit.  Lessons  learned  can  be  based  upon  positive  experiences  that 
prevent  accidents  or  save  money  or  on  negative  experiences  that  result 
in  undesirable  outcomes.  However,  if  an  organization  focuses  only  on 
failures,  its  overall  program’s  effectiveness  will  be  reduced  and  it  will 
miss  opportunities  to  improve  all  its  processes. 

•  The  verification  process  serves  to  verify  the  correctness  and 
applicability  of  lessons  submitted.  Domain  or  subject  matter  experts 
may  be  involved  in  coordinating  and  conducting  reviews  to  determine 
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whether  or  not  a  lesson  is  relevant  across  many  other  projects,  is 
unique  to  a  particular  department  or  project,  or  applies  globally  to  the 
organization  as  a  whole. 

•  The  storage  aspect  of  lessons  learned  usually  involves  incorporating 
lessons  into  an  electronic  database  for  the  dissemination  and  sharing  of 
information.  Information  should  be  stored  in  a  manner  that  allows 
users  to  identify  applicable  information  searches.  In  addition,  each 
program  should  include  a  keyword  and  functional  category  search 
capability  to  facilitate  information  retrieval. 

•  The  final  element,  and  the  most  important,  is  the  dissemination  of 
lessons  learned,  since  lessons  are  of  little  benefit  unless  they  are 
distributed  and  used  by  people  who  will  benefit  from  them. 
Dissemination  can  include  the  revision  of  a  work  process,  training,  and 
routine  distribution  via  a  variety  of  communication  media.  Lessons  can 
be  “pushed,”  or  automatically  delivered  to  a  user,  or  “pulled”  in 
situations  where  a  user  must  manually  search  for  them.  Lessons  can 
also  be  disseminated  with  an  assigned  priority  descriptor,  which 
denotes  the  risk,  immediacy,  and  urgency  of  the  lessons  learned 
content. 


Objectives,  Scope, 
and  Methodology 


The  Chairman  and  the  Ranking  Minority  Member,  Subcommittee  on  Space 
and  Aeronautics,  House  Committee  on  Science,  requested  that  we 
examine  whether  NASA  has  sufficient  mechanisms  in  place  to  ensure  that 
past  lessons  learned  are  applied  effectively  towards  future  mission 
success.  Specifically,  we  (1)  identified  what  policies,  procedures,  and 
systems  NASA  has  in  place  for  lessons  learning,  (2)  assessed  how 
effectively  these  policies,  procedures,  and  systems  facilitate  lessons 
learning,  and  (3)  determined  whether  further  efforts  are  needed  to 
improve  lessons  learning. 

We  reviewed  NASA  policies  and  procedures  and  interviewed  agency  staff 
to  determine  how  they  captured,  processed,  and  used  lessons  learned  in 
their  programs  and  projects.  We  also  obtained  briefings  and  documents  on 
the  different  mechanisms  used  by  NASA  to  capture  and  disseminate 
lessons  learned.  In  addition,  we  conducted  site  visits  at  NASA 
headquarters,  Washington,  D.C.;  Goddard  Space  Flight  Center,  Greenbelt, 
Maryland;  Kennedy  Space  Center,  Florida;  Langley  Research  Center, 
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Hampton,  Virginia;  and  the  Jet  Propulsion  Laboratory,  Pasadena, 
California.® 

To  assess  how  effectively  NASA’s  policies  and  procedures  facilitate 
lessons  learning,  we  conducted  a  survey  of  NASA  program  and  project 
managers.  We  issued  an  interim  report  in  September  2001  on  the  results  of 
our  survey.^  NASA  officials  generally  agreed  with  the  survey  results  and 
indicated  that  the  agency  must  do  a  better  job  of  communicating  various 
lessons  learning  resources  to  employees,  improving  linkages  among  the 
sources,  and  encouraging  training  to  increase  lessons  learning.  We  build 
on  the  findings  and  results  of  our  survey  in  this  report.  The  methodology 
we  employed  in  conducting  this  survey  included  the  following  steps: 

•  To  obtain  the  views  of  NASA  program  and  project  managers  on  lessons 
learned  processes  and  systems  at  NASA,  we  conducted  a  survey  in 
June  and  July  2001  using  a  self-administered  electronic  questionnaire 
posted  on  the  World  Wide  Web.  The  survey  contained  four  groups  of 
questions  on  (1)  how  NASA  personnel  collect,  access,  and  use  lessons 
learned;  (2)  the  strengths  and  weaknesses  of  current  NASA  lessons 
learned  processes,  procedures  and  systems,  including  the  Lessons 
Learned  Information  System;  (3)  potential  challenges  or  barriers  to 
sharing  lessons  learned  within  NASA;  and  (4)  ways  to  improve  the 
lessons  learned  processes  and  systems  at  NASA. 

•  In  designing  the  questionnaire,  we  interviewed  NASA  officials  and 
program  and  project  managers,  as  well  as  other  government  and 
industry  officials  who  had  insight  into  lessons  learned  processes  and 
systems.  We  also  obtained  and  reviewed  NASA  documents  and 
guidance  pertinent  to  lessons  learned.  To  further  guide  the 
development  of  appropriate  questions,  we  reviewed  current  literature 
on  lessons  learned  and  knowledge  management.  To  validate  the 
content  and  structure  of  the  questionnaire,  we  submitted  it  to  officials 
at  NASA  headquarters  and  others  for  review  and  comment.  To  verify 
the  clarity,  length  of  time  of  administration,  and  suitability  of  the 
questions,  we  also  pre-tested  the  questionnaire  with  selected  program 
and  project  managers  at  Goddard  Space  Flight  Center. 


®  The  Jet  Propulsion  Laboratory  is  a  federally  funded  research  and  development  center.  In 
our  report,  we  treat  the  Jet  Propulsion  Laboratory  as  a  NASA  center  in  our  discussions. 

^TT.S.  Oeneral  Accounting  Office,  Survey  of  NASA’s  Lessons  Learned  Process, 
GAO-01-1015R  0  Washington,  D.C.:  2001). 
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•  NASA  officials  provided  us  with  a  list  of  the  e-mail  addresses  of  all 
NASA  program  and  project  managers  as  of  April  2001.  On  June  5,  2001, 
we  sent  e-mail  messages  to  each  of  the  199  NASA  managers  in  this 
survey  population,  notifying  them  of  the  survey  and  asking  them  to 
complete  the  questionnaire.  Subsequently,  we  discovered  one 
additional  NASA  manager  who  was  eligible  for  our  survey,  and 
removed  seven  managers  who  were  ineligible  for  the  survey  because 
they  were  not  project  or  program  managers  at  the  time  of  the  survey. 
We  also  eliminated  one  duplicate  listing,  resulting  in  a  final  survey 
population  of  192. 

•  Over  the  following  several  weeks,  until  closing  the  survey  on  July  13, 
2001,  we  received  a  total  of  115  useable  responses,  for  an  overall 
response  rate  of  60  percent.  All  of  the  responses  had  been  submitted 
using  the  Web  questionnaire,  although  one  participant  had  asked  to 
submit  a  paper  version  of  the  survey,  due  to  problems  in  accessing  the 
Web  survey.  Of  the  77  non-respondents,  9  provided  partial 
questionnaire  responses  but  had  not  indicated  that  they  were  finished 
with  the  questionnaire.  Partial  responses  were  not  included  in  our 
survey  results. 

While  we  believe  that  our  survey  results  are  generalizable  to  the 
population  of  NASA  program  and  project  managers  as  described  above, 
the  practical  difficulties  of  conducting  any  survey  may  introduce  errors 
into  estimates  made  from  surveys.  Although  we  administered 
questionnaires  to  all  known  members  of  the  population,  and  thus  our 
results  are  not  subject  to  sampling  error,  non-response  to  the  entire  survey 
or  individual  questions  can  introduce  a  similar  type  of  variability  or  bias 
into  our  results  to  the  extent  that  those  not  responding  differ  from  those 
who  do  respond  in  how  they  would  have  answered  our  survey  questions. 

In  addition,  population  coverage  errors  can  occur  if  some  members  of  the 
population  are  excluded  from  the  survey.  Measurement  errors  can  arise 
from  how  questions  are  interpreted  by  respondents  and  mistakes  made  by 
respondents.  Data  processing  errors  can  arise  during  the  handling  or 
analysis  of  responses.  We  took  steps  in  the  design,  data  collection,  and 
analysis  phases  of  our  survey  to  minimize  such  errors,  such  as  pre-testing 
questionnaires  before  the  survey,  following  up  with  those  not  reachable  at 
original  e-mail  addresses  or  otherwise  not  immediately  responding,  and 
checking  for  errors  in  computer  programming  used  to  analyze  survey 
results.  In  addition,  the  distribution  of  respondents  across  NASA 
enterprise  areas  generally  reflected  the  actual  distribution  of  the  entire 
population,  which  was  consistent  with  our  belief  that  non-response  error 
was  not  significant. 
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To  identify  practices  for  improving  NASA’s  lessons  learning  processes  and 
systems,  we  reviewed  existing  literature  on  lessons  learning  and 
knowledge  management  practices.  In  addition,  we  identified  a  selected 
number  of  commercial  and  government  organizations  that  are 
practitioners  of  knowledge  management  and  obtained  information  related 
to  their  experience.  In  making  our  selections,  we  reviewed  literature  and 
spoke  with  industry  and  academic  experts  to  find  organizations 
recognized  for  their  ability  to  share  lessons  learned  or  effectively  manage 
knowledge.  We  identified  the  following  organizations:  World  Bank,  United 
States  Department  of  Energy,  Ford  Motor  Company,  TRW,  Boeing  Space 
and  Communications,  Northrop  Grumman,  and  Lockheed  Martin. 

We  discussed  with  representatives  from  each  organization  the  processes 
they  used  for  sharing  information,  emphasizing  those  practices  that  were 
critical  for  successful  lessons  learning.  During  our  discussions  with  the 
organizations,  we  compared  and  contrasted  their  practices  with  those  of 
NASA.  This  allowed  us  to  identify  practices  that  NASA  could  use  to 
improve  its  processes  and  systems.  For  each  organization,  we  interviewed 
key  managers  and  obtained  documentation  to  determine  (1)  the  processes 
and  tools  used  to  enhance  information  sharing  and  (2)  the  extent  to  which 
these  processes  and  tools  affected  the  organizations’  ability  to  share 
information. 

Our  work  was  performed  from  October  2000  through  September  2001  in 
accordance  with  generally  accepted  government  auditing  standards. 
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NASA  uses  various  mechanisms  to  communicate  lessons  garnered  from 
past  programs  and  projects.  Policies  and  guidelines,  programmatic  and 
technical  reviews,  mentoring  and  training  programs,  the  Academy  of 
Program  and  Project  Leadership,  and  the  Lessons  Learned  Information 
System  (LLIS)  are  all  mechanisms  employed  by  NASA  for  capturing  and 
sharing  lessons  learned.  While  not  the  only  means  available  for  capturing 
and  disseminating  lessons  learned,  LLIS  is  the  “official”  agency-wide 
repository  for  such  lessons.  In  addition  to  agency- wide  mechanisms  for 
collecting  and  sharing  lessons  learned,  several  NASA  centers  also 
operated  lessons  learned  systems  configured  to  support  specific  programs, 
such  as  the  International  Space  Station. 

To  improve  the  way  the  agency  captures  and  shares  information  and 
lessons  learned,  NASA  has  recently  formed  a  knowledge  management 
team.  Knowledge  management  is  a  business  strategy  used  by  many 
organizations  that  strive  to  make  more  effective  use  of  the  experience  and 
expertise  of  employees  within  an  organization.  Organizations  believe  that 
by  developing  new  ways  to  capture  and  share  knowledge,  they  can 
increase  productivity,  collaboration,  and  innovation. 


NASA’s  Policies  and 
Procedures  Require 
Lessons  Learning 


NASA’s  policies  and  procedures  require  the  continuous  capture, 
dissemination,  and  utilization  of  lessons  learned.  NASA  defines  a  lesson 
learned  as  knowledge  or  understanding  gained  by  experience.  The 
experience  may  be  positive,  as  in  a  successful  test  or  mission,  or  negative, 
as  in  a  mishap  or  failure.  A  lesson  must  be  significant  in  that  it  has  a  real  or 
assumed  impact  on  operations;  valid  in  that  it  is  factually  correct;  and 
applicable  in  that  it  identifies  a  specific  design,  process,  or  decision  that 
reduces  or  eliminates  the  potential  for  failures  and  mishaps,  or  reinforces 
a  positive  result. 


Program  and  Project  To  ensure  that  lessons  learned  from  previous  experiences  are  used  as  a 

Managers  Are  Directed  to  resource,  NASA  Procedures  and  Guidelines  for  Program  and  Project 
Review  Lessons  Learned  Management  Processes  and  Requirements  (NPG  7120.5A)  directs  each 

program  and  project  manager  to  review  and  apply  significant  lessons 
learned  from  the  past  throughout  the  program  or  project  life  cycle. 
Managers  are  also  directed  to  consult  LLIS  prior  to  major  milestones  to 
gain  lessons  from  past  programs  and  projects  that  are  documented  and 
collected  as  a  benefit  to  future  programs  and  projects.  Program  and 
project  managers  are  also  directed  to  document  and  submit  to  the  LLIS  in 
a  timely  manner  any  significant  lessons  learned  throughout  the  life  of  a 
project  or  program.  One  of  the  objectives  of  the  agency’s  guidance  is  to 
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protect  against  the  recurrence  of  past  mistakes  and  provide  a  mechanism 
for  sharing  best  practices. 

NASA  Mishap  Reporting  and  Investigating  Policy  (NPD  8621. IG)  and 
NASA  Procedures  and  Guidelines  for  Mishap  Reporting,  Investigating,  and 
Recordkeeping  (NPG  8621.1)  also  require  the  development  of  lessons 
learned  from  the  mishaps  for  possible  application  to  existing  or  future 
programs.^  In  addition,  NASA  is  supposed  to  exchange  lessons  learned  and 
other  pertinent  safety  information  of  common  interest  with  other  federal 
agencies,  international  partners,  and  NASA  contractors.  Since  NASA  uses 
contractors  to  produce  many  of  its  systems,  NASA’s  Procedures  and 
Guidelines  for  Management  of  Government  Safety  and  Mission  Assurance 
Surveillance  Functions  for  NASA  Contracts  (NPG  8735.2)  requires  that 
lessons  learned  during  the  contract  surveillance  activities  be  captured  and 
submitted  to  LLIS. 


NASA  Uses  Numerous 
Methods  for  Lessons 
Learning 


Lessons  learned  mechanisms  are  used  by  many  commercial  and 
government  organizations  to  share  and  use  knowledge  derived  from 
experience  to  (1)  promote  the  recurrence  of  desirable  outcomes,  or  (2) 
preclude  the  recurrence  of  undesirable  outcomes.  Policies  and  guidelines, 
programmatic  and  technical  reviews,  training  programs,  and  the  LLIS  are 
all  mechanisms  employed  by  NASA  for  capturing  and  sharing  lessons 
learned.  While  not  the  only  means  available  for  capturing  and 
disseminating  lessons  learned,  LLIS  is  NASA’s  only  agency-wide  repository 
for  such  lessons. 


LLIS  is  NASA’s  Principal  The  purpose  of  LLIS  is  to  capture  and  share  lessons  learned  from 

Mechanism  for  Sharing  programs,  projects,  and  missions  and  to  help  ensure  that  NASA  does  not 

Lessons  Learned  have  to  keep  “relearning”  the  lessons  of  the  past.  LLIS  was  created  in  1995 

and  subsequently  updated  to  include  a  Web  interface.  Program  and  project 
managers  are  directed  to  use  LLIS  to  identify  and  share  lessons  learned, 
thus  facilitating  the  early  incorporation  of  safety,  reliability, 
maintainability,  and  quality  into  the  design  of  flight  and  ground  support 
hardware,  software,  facilities,  and  procedures.  Currently,  the  system 


^  NASA  defines  a  mishap  as  “an  unplanned  event  that  results  in  injury  to  non-NASA 
personnel  caused  by  NASA  operations;  damage  to  pubhc  or  private  property  (including 
foreign  property)  caused  by  NASA  operations;  occupational  injury  or  occupational  illness 
to  NASA  personnel;  damage  to  NASA  property  caused  by  NASA  operations;  or  mission 
failure.” 
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contains  over  900  lessons  learned  on  the  development  and  design  of 
aeronautics  and  space  systems. 

Lessons  entered  in  the  LLIS  database  are  screened  for  relevance  and  to 
ensure  that  they  do  not  contain  sensitive  or  proprietary  information.  Initial 
reviews  of  lessons  are  usually  conducted  by  the  centers  with  a  final  review 
conducted  by  the  Office  of  Safety  and  Mission  Assurance.  After  a  lesson  is 
entered  into  the  system,  it  is  not  reviewed  for  currency  or  relevance.  In 
other  words,  once  lessons  are  entered,  they  remain  in  the  database 
indefinitely.  The  Office  of  Safety  and  Mission  Assurance  monitors  the 
system  to  determine  the  number  of  times  it  is  accessed,  a  search  is 
conducted,  and/or  search  results  are  obtained,  but  does  not  collect 
information  on  the  number  of  times  lessons  are  applied  to  a  program  or 
project. 


Administrator  for  Safety 
and  Mission  Assurance  is 
Responsible  for  LLIS 


NASA’s  Quality  Management  System  Policy  (NPD  8730.3)  establishes  the 
Quality  Management  Systems  used  by  the  agency  to  identify  and  control 
processes  to  assure  the  quality  of  hardware,  software,  and  services 
provided.^  According  to  this  policy,  the  Associate  Administrator  for  Safety 
and  Mission  Assurance  is  responsible  for  ensuring  that  the  LLIS  database 
is  maintained  and  accessible.  Furthermore,  the  policy  states  that  the 
associate  deputy  administrator  and  the  center  directors  are  responsible  for 
ensuring  that  LLIS  is  used  to  ensure  quality  and  document,  investigate,  and 
apply  lessons  learned  for  all  programs  and  projects. 


Center  and  Program  Based 
Systems  Used  to  Collect  and 
Disseminate  Lessons  Learned 


While  NASA’s  LLIS  is  supposed  to  be  the  NASA-wide  system,  several 
NASA  centers  maintain  their  own  electronic  lessons  learned  systems.  For 
example,  the  Goddard  Space  Flight  Center  uses  the  Flights  Programs  and 
Projects  Directorate  Lessons  Learned  Database  to  collect  lessons  learned 
for  each  flight  project.  Once  a  lesson  is  approved,  it  is  passed  along  to  the 
center  data  manager  for  review  and  eventual  submission  to  LLIS,  thereby 
ensuring  that  what  has  been  learned  at  the  Goddard  Space  Flight  Center  is 
accessible  to  other  facilities.  One  lesson  that  was  passed  along  from  the 
Goddard  Space  Flight  Center  system  is  the  benefit  of  testing  deployable 
devices,  such  as  the  Landsat  7  solar  array,  early  in  satellite  development. 
Testing  devices  early  is  a  cost-effective  tactic  for  developing  complex  and 
critical  mechanism  subsystems.  By  testing  the  Landsat  7  solar  array  early. 


^  NASA’s  Quality  Management  System  is  implemented  in  accordance  with  the  International 
Organization  for  Standardization’s  ISO  9000  Quahty  Management  System  Standards. 
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program  officials  were  able  to  discover  problems  that  could  have  resulted 
in  a  program  delay  and  increased  costs.  Another  lesson  in  the  Flights 
Programs  and  Projects  Directorate  Lessons  Learned  Database  makes  a 
recommendation  on  minimizing  the  effects  associated  with  ongoing 
construction  at  mission-critical  facilities.  Specifically,  NASA  officials 
learned  the  importance  of  developing  a  strategy  to  minimize  conflicts 
between  the  construction  schedule  and  project  requirements.  One  of  the 
recommendations  for  future  projects  is  that  construction  activities  in 
mission-specific  support  areas  of  a  mission-critical  facility  should  be 
suspended  30  days  prior  to  launch  to  ensure  that  the  facility  is  ready  to 
support  mission  operations. 

In  addition,  program-based  lessons  learned  databases  have  been 
established  for  key  NASA  programs.  For  example,  the  International  Space 
Station  and  Phase  1  Lessons  Learned  Database  documents  historical 
lessons  learned  from  the  Phase  1  Program,  U.S./Russian  Program  Office  of 
the  International  Space  Station  Program;  the  top  ten  lessons  learned 
during  the  Skylab  program;  and  current  lessons  learned  from  the 
International  Space  Station  Program.^  The  International  Space  Station  and 
Phase  1  Lessons  Learned  Database  is  the  official  system  for  submitting 
new  lessons  learned  in  the  International  Space  Station  program.  The 
lessons  learned  collected  in  that  Lessons  Learned  Database  are  intended 
to  be  used  as  tools  for  the  improvement  of  International  Space  Station 
operations.  NASA  officials  are  deliberating  the  feasibility  of  transferring 
the  lessons  learned  from  the  International  Space  Station  database  to  the 
NASA-wide  LLIS. 

Another  mechanism  used  by  NASA  to  share  lessons  is  training.  NASA’s 
Academy  of  Program  and  Project  Leadership  (APPL)  provides  numerous 
courses  throughout  the  year.''  Many  of  the  classes  employ  case  studies  to 
learn  from  successes  and  failures.  According  to  an  academy  official, 
lessons  learned  are  incorporated  into  the  different  courses.  Lessons 
incorporated  into  the  curriculum  are  generic  and  can  be  applied  to  ah 
programs  (i.e.,  better  communications)  rather  than  technical  issues  unique 
to  a  particular  program.  About  4,000  program  management  employees 


^  Skylab  was  the  United  States’  first  experimental  space  station.  The  Skylab  program’s 
objectives  were  to  prove  that  humans  could  live  and  work  in  space  for  extended  periods, 
and  to  expand  our  knowledge  of  solar  astronomy. 

The  Academy  of  Program  and  Project  Leadership  is  the  NASA  education  organization 
responsible  for  training  program  and  project  managers. 
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have  attended  academy  courses,  and  thus  they  contribute  to  sharing 
information  within  the  organization. 

In  addition  to  formal  training,  APPL  engages  in  development  activities, 
which  play  a  role  in  diffusing  knowledge  throughout  NASA.  For  example, 
it  hosts  a  series  of  forums  on  “hot  topics”  to  gather  and  disseminate 
information.  Another  activity  involves  capturing  and  sharing  the 
experiences  of  program  managers.  Once  a  year,  APPL  sponsors  two 
sessions  in  which  project  managers  considered  the  “best”  by  their  centers 
are  sent  to  discuss  project  management.  At  these  sessions,  two  or  three 
project  managers  from  each  center  share  what  works  and  the  types  of 
tools  that  they  need  to  accomplish  their  jobs. 

Another  activity  used  by  APPL  is  encouraging  senior  program  managers  to 
share  their  knowledge  through  a  series  of  short  stories  made  available 
through  its  Web  site.  Each  story  discusses  a  topic  that  will  help  program 
managers  succeed.  One  recently  completed  story  dealt  with  the 
importance  of  saying  no.  According  to  a  NASA  official,  many  developing 
project  and  program  managers  are  driven  by  NASA’s  “can  do”  attitude  and 
have  a  hard  time  knowing  when  to  say  “no”  during  a  project.  NASA  is  also 
identifying  capable  individuals  to  mentor  new  project  and  program 
managers.  In  addition,  managers  are  provided  several  online  tools  and 
training  resources,  such  as  project  management  tools  that  can  be  used  to 
effectively  manage  a  project. 

Changes  to  Policy  Also  Reflect  According  to  NASA  officials,  periodic  revisions  to  agency  policies  and 

Lessons  Learned  procedures  are  another  way  that  the  agency  shares  lessons  learned.  For 

example,  NASA  is  currently  revising  its  Program  and  Project  Management 
Processes  and  Requirements  (NPG  7120.5A)  and  will  incorporate  some  of 
the  issues  raised  by  the  Mars  Climate  Orbiter  and  Polar  Lander  mishaps. 
This  document  is  intended  to  provide  the  basic  processes,  requirements, 
and  responsibilities  for  managing  NASA  programs  and  projects. 


Additional  Steps 
Taken  by  NASA  to 
Facilitate  Lessons 
Learning 


In  response  to  the  Mars  Program  failures  and  the  recommendations  of  the 
NASA  Integrated  Action  Team,  NASA  has  taken  steps  to  promote  the 
capture  and  sharing  of  knowledge  within  the  agency  by  developing  new 
policies,  processes,  and  practices  for  program  and  project  management. 
Foremost  among  these  is  the  development  of  a  business  strategy 
commonly  referred  to  as  knowledge  management.  According  to  NASA 
officials,  knowledge  management  has  the  potential  to  link  people  with  the 
information  and  resources  they  need  to  complete  tasks  faster,  better,  and 
cheaper.  Knowledge  management  can  be  defined  as  the  way  that 
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organizations  create,  capture,  and  re-use  knowledge  to  achieve 
organizational  objectives.  Organizations  that  manage  knowledge 
effectively  claim  higher  performance,  innovation,  and  collaboration  among 
employees.  Lesson  learning  in  the  context  of  knowledge  management  is 
defined  as  “knowledge,  both  positive  and  negative,  gained  through 
experience,  which  if  shared,  would  benefit  the  work  of  others.” 


NASA  Created  a 
Knowledge  Management 
Team  to  Coordinate 
Knowledge  Sharing 
Activities 


NASA  has  recently  created  a  knowledge  management  team,  composed  of 
headquarters  and  center  officials,  to  advise  the  agency’s  chief  information 
officer  (CIO).  The  team  does  not  act  as  a  central  authority  for  knowledge 
management,  but  rather  as  a  focal  point  for  coordinating  knowledge 
management  activities  conducted  throughout  the  agency.  Its  goal  is  to 
connect  the  various  knowledge  management  activities  ongoing  in  the 
agency  and  avoid  duplication.  The  team  is  currently  working  on  efforts  to 
baseline  knowledge  management  in  NASA,  gather  requirements,  and 
define  opportunities  for  applying  knowledge  management. 


Knowledge  Management  Several  centers  have  initiated  knowledge  management  projects  in  the  past 
Projects  Conducted  by  few  years.  For  example,  the  Ames  Research  Center  is  designing  and 

Selected  Centers  developing  a  comprehensive  information  technology  program  called 

Design  for  Safety/Engineering  for  Complex  Systems  to  help  programs  and 
projects  agency-wide  manage  risk  and  capture  and  disseminate  knowledge 
and  lessons  learned.  The  Johnson  Space  Center  is  piloting  a  Quality 
Assurance  database,  and  the  Jet  Propulsion  Laboratory  is  developing  an 
“experts”  directory  as  well  as  a  Technical  Questions  database.  In  addition, 
the  Kennedy  Space  Center  has  created  a  category  of  knowledge  managers 
in  each  of  its  directorates  who  are  responsible  for  identifying  and 
recognizing  core  knowledge  management  competencies.  Furthermore,  the 
Langley  Research  Center  has  initiated  a  process  to  reward  scientists  if 
knowledge  they  have  shared  is  used  by  others.  The  re-use  of  the 
knowledge  must  be  documented  in  order  for  the  scientist  to  receive  the 
reward. 


ClO-Sponsored  Knowledge  NASA’s  CIO  supports  agency  knowledge  management  efforts  by  investing 
Management  Pilot  Projects  fe  information  technology  projects  to  facilitate  knowledge  sharing.  The 

NASA  Knowledge  Management  Team  advises  the  CIO  on  projects 
submitted  by  the  centers  for  funding.  In  fiscal  year  2000,  the  CIO  funded 
three  knowledge  management  pilot  projects  totaling  $567,000.  For  fiscal 
year  2002,  the  CIO  has  a  budget  of  $3  million  that  can  be  used  to  fund 
selected  knowledge  management  initiatives.  Because  knowledge 
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management  projects  are  not  centrally  managed  or  funded,  there  is  no 
visibility  over  how  much  the  various  NASA  centers  spend  on  them. 

One  of  the  projects  being  funded  is  to  develop  an  agency-wide  Web  portal 
to  bring  together  NASA’s  online  information  resources,^  which  are 
currently  dispersed  across  different  agency  functional  areas,  centers,  and 
programs.  The  portal  will  integrate  existing  resources  and  provide  users 
with  more  direct  access  to  information  from  across  the  agency.  A  second 
project  is  intended  to  develop  a  directory  of  agency  expertise,  which  will 
allow  NASA  personnel  to  quickly  locate  others  working  in  a  related  field 
or  on  a  particular  project  and  thus  enhance  collaboration  among 
distributed  groups.  When  complete,  the  directory  will  be  integrated  into 
the  NASA  Web  portal. 

The  third  ClO-funded  project  will  improve  the  capture  and  re-use  of 
lessons  learned  by  augmenting  the  agency’s  current  LLIS.  The  pilot  project 
was  a  collaborative  effort  involving  the  Jet  Propulsion  Laboratory,  Langley 
Research  Center,  and  the  Air  Force  Research  Laboratory.  The  Jet 
Propulsion  Laboratory  team  was  responsible  for  defining  the  processes 
used  to  create  and  capture  lessons  learned,  as  well  as  the  functional  and 
system  requirements  for  the  system’s  redesign.  The  Langley  Center  was 
responsible  for  developing  the  digital  library  technology  for  the  system. 
The  Air  Force  Research  Laboratory  role  was  to  evaluate  the  system  design. 

After  spending  about  $135,000  to  develop  a  system  requirements  document 
and  a  new  system  software  design,  NASA  terminated  the  pilot  project. 
According  to  a  NASA  official,  the  Office  of  Safety  and  Mission  Assurance 
considered  the  functional  and  system  requirements  document  to  be 
inadequate,  and  their  office  lacked  the  resources  to  implement  the 
proposed  prototype.  However,  since  the  termination  of  the  pilot,  the 
Goddard  Space  Flight  Center  has  been  tasked  with  determining  ways  to 
improve  the  LLIS.  Officials  there  have  proposed  a  subscription  service  that 
would  enable  users  to  receive  lessons  learned  tailored  to  their  interests. 
Other  follow-on  LLIS  improvement  efforts  under  consideration  include  an 
improved  searching  and  storage  capability  and  the  ability  to  append  to, 
update,  and  control  lessons  configuration  over  extended  periods  of  time. 


^  A  portal  is  a  Web  site  that  is  a  major  starting  site  for  users  when  they  connect  to  the 
internet  or  Intranet.  A  portal  can  include  features  such  as  a  topical  index,  search  engine, 
e-mail,  news,  and  links  to  other  onhne  tools. 
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Strategic  Plan  for 
Knowledge  Management 


In  March  2001,  the  Knowledge  Management  Team  issued  a  Strategic  Plan 
for  Knowledge  Management  that  provides  a  framework  for  addressing 
knowledge  management  in  NASA  in  terms  of  people,  process,  and 
technology  (see  table  2).  According  to  the  plan,  “people”  are  the  most 
important  component  of  knowledge  management.  It  calls  for  recognizing 
and  rewarding  people  for  sharing  knowledge  as  well  as  encouraging  story 
telling  and  establishing  communities  of  practice  to  promote  knowledge 
sharing.  The  plan  also  advocates  the  development  of  policies  and 
procedures  for  capturing  and  managing  knowledge  that  must  be  published 
and  understood  by  users.  In  addition,  the  plan  emphasizes  the  need  for 
better  information  technology  to  facilitate  knowledge  management.  In  the 
context  of  lessons  learning,  the  plan  suggests  providing  a  subscription 
service  that  tailors  specific  lessons  to  user-defined  interests  and  pushing 
positive  lessons  to  users’  desktops  based  upon  project  tasks  or 
organizational  characteristics. 


Table  2:  NASA’s  Framework  for  Knowledge  Management 


People 

Process 

Technology 

Enable  remote 

Enhance  knowledge 

Enhance  system  integration 

collaboration 

capture 

and  data  mining 

Support  communities  of 
practice 

Reward  and  recognize 
knowledge  sharing 

Encourage  storytelling 

Manage  information 

Utilize  intelligent  agents 

Exploit  expert  systems 

Source:  Strategic  Plan  for  Knowledge  Management,  March  2001. 
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Although  NASA  recognizes  it  is  more  critical  than  ever  to  share  knowledge 
and  collect,  store,  and  distribute  lessons  learned,  the  agency’s  current 
array  of  processes,  systems,  and  culture  do  not  facilitate  such  an 
environment.  Importantly,  our  survey  revealed  that  lessons  are  not 
routinely  identified  and  shared  by  program  and  project  managers,  LLIS  has 
not  proven  useful  to  these  managers,  and  there  is  little  incentive  for 
managers  to  share.  As  a  result,  program  and  project  managers  are  not 
maximizing  opportunities  to  apply  lessons  learned  toward  its  faster, 
better,  and  cheaper  efforts. 


Fundamental 
Weaknesses  Exist  in 
NASA’s  Processes, 
Procedures,  and 
Systems  for  Lesson 
Learning 


NASA’s  procedures  and  guidelines  require  that  program  and  project 
managers  review  and  apply  lessons  learned  from  the  past  throughout  a 
program’s  or  project’s  life  cycle  and  document  and  submit  any  significant 
lessons  learned  in  a  timely  manner.  However,  a  survey  we  conducted  of  all 
NASA  program  and  project  managers  revealed  fundamental  weaknesses 
agency-wide  in  the  collection  and  sharing  of  lessons  learned.  While  some 
lessons  learning  does  take  place,  our  survey  identified  that  lessons  are  not 
routinely  identified,  collected,  or  shared  by  program  and  project 
managers.  In  addition,  many  respondents  indicated  that  they  are 
dissatisfied  with  NASA’s  lessons  learned  processes  and  systems. 
Respondents  also  identified  challenges  or  cultural  barriers  to  the  sharing 
of  lessons  learned  as  well  as  areas  of  improvement.  As  a  consequence, 
there  is  no  assurance  that  lessons  are  being  learned  and  applied  to  future 
mission  success.  We  provided  the  basic  results  of  our  survey  in  a 
September  2001  report.^  The  following  analysis  builds  on  those  results. 


Limited  Sharing  of  Lessons  To  better  understand  the  extent  that  lessons  are  being  collected  and 
Learned  Agency- Wide  shared  within  NASA,  we  asked  program  and  project  managers  what  they 

knew  about  lessons  generated  by  their  own  programs  and  centers  as  well 
as  by  other  centers.  As  shown  in  figure  2,  program  and  project  managers 
responded  that  while  they  are  very  or  somewhat  knowledgeable  about 
lessons  generated  by  their  own  programs  and  centers,  they  know  less 
about  lessons  generated  elsewhere.  This  fact  was  further  substantiated  by 
survey  results  which  show  that  managers  primarily  identify  lessons 
through  program-  or  center-based  activities  such  as  project  reviews  or 
informal  discussions  with  colleagues.  For  example,  one  project  manager 


^  U.S.  Gonoral  Ancounting  Office,  Survey  of  NASA’s  Lessons  Learned  Process, 
GAO-01-1015R  0  Washington,  D.C.:  2001). 
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stated  that  monthly  reviews  with  center  management  provided  a  way  to 
capture  and  share  lessons  learned.  The  LLIS,  NASA’s  primary  method  for 
disseminating  lessons  learned  agency-wide,  was  not  identified  as  a  primary 
source  for  lessons  learning. 


Figure  2:  Program  and  Project  Managers  Knowledge  of  Lessons  Learned 

100 

80 

60 

40 

20 

0 

Own  Program  Programs  and  Projects  in  Own  Center  Programs  and  Projects  in  Other  Centers 


Very  Knowledgeable 
Somewhat  Knowledgeable 
Not  Very  Knowledgeable 
No  Answer 


Respondents  to  our  survey  also  indicated  that  their  principal  sources  for 
identifying  lessons  learned  were  (1)  system  and  engineering  reviews,  (2) 
program  and  project  briefings,  and  (3)  informal  discussions  with 
colleagues.  According  to  NASA  program  officials,  regular  program  and 
project  meetings,  reviews,  and  briefings  provide  an  opportunity  to  share 
information,  including  lessons  learned.  For  example,  officials  at  Kennedy 
Space  Center  explained  that  after  each  space  shuttle  launch  they  gather  to 
discuss  what  went  well  and  what  could  be  done  better.  Program  and 
project  managers  also  indicated  that  they  maintain  informal  networks  with 
colleagues  where  lessons  learning  take  place. 

Project  reviews  and  informal  discussions  with  colleagues  are  important 
mechanisms  for  lessons  learning;  however,  they  do  not  facilitate  the 
systematic,  agency-wide  sharing  of  lessons.  LLIS,  which  was  established 
for  such  agency-wide  sharing,  is  not  widely  used,  according  to  survey 
respondents,  even  though  NASA  guidelines  state  that  program  and  project 
managers  should  consult  it  prior  to  major  milestones  and  submit  any 
significant  lessons  learned  in  a  timely  manner.  One  reason  the  system  is 
not  widely  used,  according  to  one  center  official,  is  that  its  lessons  cover 
so  many  topics  that  it  is  difficult  to  search  for  an  applicable  lesson.  One  of 
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our  survey  respondents  indicated  that  “It  is  difficult  to  weed  through  all 
the  irrelevant  lessons  to  get  to  the  few  ‘jewels’  that  you  need  to  find.  There 
should  be  better  categories  to  find  relevant  lessons.”  Also,  contractors, 
who  perform  much  of  NASA’s  work,  are  unable  to  query  the  system  easily. 
As  shown  in  figure  3,  in  the  past  2  years,  43  percent  of  program  and  project 
managers  have  not  submitted  a  lesson  to  the  LLIS  compared  to  23  percent 
of  managers  who  have.  The  survey  revealed  there  is  also  a  low  level  of 
awareness  of  the  system  among  managers;  27  percent  of  program  and 
project  managers  were  not  aware  of  LLIS  before  our  survey. 

According  to  some  project  officials,  accessing  LLIS  is  time  consuming  and 
uses  significant  project  resources.  One  official  said  it  took  about  two 
weeks  to  review  the  lessons  in  the  database.  Another  said  that  developing 
the  process  of  assessing  the  applicability  of  lessons  learned  from  the 
database  for  that  project  was  very  labor  intensive  and  costly.  For  example, 
one  project  reviewed  all  the  lessons  learned  in  LLIS  and  attached  the 
appropriate  lessons  learned  as  an  appendix  to  the  project’s  Mission 
Assurance  Plan.  In  contrast,  another  project  developed  a  matrix 
containing  all  the  lessons  learned  from  LLIS  categorized  by  functional  area 
and  technical  discipline.  Project  teams  organized  by  discipline  then 
reviewed  the  lessons  learned  for  applicability  to  the  project.  Further,  while 
some  projects  formally  reviewed  lessons  learned  from  the  Mars  Program 
failures,  others  did  not.  However,  other  projects  reviewed  the  lessons 
learned  from  both  LLIS  and  from  the  Mars  Program  failures. 
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Figure  3:  Many  Program  and  Project  Managers  Have  Not  Submitted  Lessons  to  the 
Lessons  Learned  Information  System 
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Dissatisfaction  with 
Lessons  Learned 
Processes  and  Systems 


Our  survey  also  determined  that  more  managers  are  very  or  generally 
dissatisfied  than  are  satisfied  with  NASA’s  lessons  learned  processes  and 
systems,  including  LLIS.  This  level  of  dissatisfaction  may  stem  from  the 
fact  that  58  percent  of  managers  stated  that  current  processes  and  systems 
do  not  allow  them  to  retrieve  the  right  lessons  at  the  right  time.  As  shown 
in  figure  4,  the  level  of  dissatisfaction  may  also  stem  from  the  fact  that  53 
percent  managers  stated  that  they  found  lessons  useful  less  than  25 
percent  of  the  time.  One  senior  center  official  stated  LLIS  serves  more  as  a 
repository  of  information  that  favors  technical  lessons,  and  is  not  effective 
for  passing  along  such  information. 

Also,  managers  stated  that  it  would  be  beneficial  if  more  emphasis  were 
placed  on  the  reporting  of  positive  lessons  that  are  derived  from 
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successful  program  outcomes.  For  example,  a  positive  lesson  noted  on  the 
Jet  Propulsion  Laboratory’s  Deep  Space  1  program  was  having  the  same 
program  management  team  from  the  beginning  through  the  end  of  the 
primary  mission.  One  manager  noted  that  positive  lessons  can  be  even 
more  helpful  to  a  project  than  negative  ones  because  even  if  a  project 
avoids  negatives,  without  positives  managers  may  not  follow  the  most 
effective  and  efficient  path  to  mission  success. 


Figure  4:  Most  Program  and  Project  Managers  Could  Not  Identify  Helpful  Lessons 
for  Their  Program  or  Project 
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Barriers  Exist  to  Lessons 
Learning 


If  lessons  learning  is  to  succeed,  it  is  important  that  the  organizational 
culture  fosters  the  value  of  sharing  knowledge  based  on  others’ 
experiences.  Our  survey  respondents  identified  several  notable  cultural 
barriers  that  affect  lessons  learning  within  NASA.  These  included: 


Page  31 


GAO-02-195  NASA 


Chapter  3:  Lessons  Learned  Are  Not  Being 
Consistently  Shared  or  Applied  Toward 
Fntnre  Program  Snccess 


•  perception  that  there  is  no  benefit  from  lessons  learned; 

•  lack  of  time  to  share  knowledge; 

•  lack  of  trust; 

•  intolerance  for  mistakes. 

Our  survey  found  that  many  program  and  project  managers  (39  percent) 
believe  that  implementing  lessons  learned  contributes  only  some  or  little 
to  improving  mission  effectiveness.  This  perception  was  highlighted  by 
one  manager  who  stated  that  “Until  we  can  adopt  a  culture  that  admits 
frankly  to  what  really  worked  and  didn’t  work,  1  find  many  of  these  tools 
to  be  suspect.” 

Another  critical  barrier  highlighted  by  the  survey  is  the  lack  of  time 
available  for  lessons  learned.  Managers  reported  that  they  and  their 
project  staff  are  very  busy  conducting  assigned  project  tasks,  and  little,  if 
any,  time  is  allotted  for  lessons  learning.  One  manager  noted  that  “It  is 
time  consuming  to  agree  on  correct  lessons  learned  for  a  project  and 
accurately  describe  the  issue. .  .in  a  way  that  is  helpful  to  a  project.” 
Managers  also  noted  that  there  is  reluctance  to  share  negative  lessons  for 
fear  that  they  might  not  be  viewed  as  good  project  managers.  According  to 
some  respondents,  this  appears  to  stem  from  a  culture  that  sees  lessons 
learned  as  negative,  i.e.,  an  admission  of  failure.  For  example,  one 
manager  noted  that  “People  are  never  rewarded  for  telling  about  how  they 
screwed-up  and  caused  a  problem/mistake. . ..  This  will  continue  to  be  a 
problem  until  a  way  is  found  to  allow  and  encourage  people  to  talk  about 
their  mistakes  without  feeling  that  they  are  risking  their  careers.”  The 
cultural  barriers  to  lessons  learning  identified  above  present  a  serious 
challenge  for  NASA  and  the  agency  may  well  be  missing  fundamental 
opportunities  to  share  and  apply  knowledge  toward  future  mission 
success. 


Suggested  Improvements  While  program  and  project  managers  identified  a  number  of  ways  for 
to  Lessons  Learning  improving  lessons  learning  within  NASA,  mentoring  was  their  first  choice. 

One  manager  noted  that  the  most  effective  lessons  learned  process  results 
from  the  passing  down  of  lessons  from  experienced  people  to  those  less 
experienced.  We  found  that  mentoring  programs  have  been  reinvigorated 
at  a  few  centers  in  response  to  lessons  learned  from  the  faster,  better,  and 
cheaper  changes  as  well  as  concerns  that  management  expertise  is  being 
lost  because  of  increasing  retirements.  Other  suggestions  for  improvement 
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included  “storytelling”  or  “after-action  reviews.”  For  example,  NASA’s 
Academy  of  Program  and  Project  Leadership  asks  senior  program 
managers  to  share  their  knowledge  through  a  series  of  short  stories.  Each 
story  is  1  to  3  pages  and  discusses  a  topic  that  will  help  program  mangers 
to  succeed.  A  recently  completed  story  deals  with  the  importance  of 
knowing  when  to  say  “no”  in  a  project.  However,  most  managers  in  our 
survey  did  not  identify  the  academy  as  a  principal  source  for  lessons 
learning.  Another  notable  suggestion  was  that  senior  management  should 
be  accountable  for  the  infusion  or  engineering  of  lessons  into  a  project 
with  some  kind  of  a  matrix/metric  to  validate  that  the  lessons  were 
actually  incorporated. 

Managers  also  suggested  a  number  of  improvements  regarding  LLIS.  As 
shown  in  table  3,  suggested  improvements  were  related  to  (1)  developing  a 
means  to  disseminate  lessons  from  users,  (2)  improving  the  search 
capability,  and  (3)  including  more  positive  lessons.  Currently  with  the 
LLIS,  there  is  no  automated  dissemination  of  lessons,  it  is  difficult  to  find 
relevant  lessons,  and  there  are  few  positive  lessons  included. 
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Table  3:  Program  and  Project  Manager  Observations  and  Suggestions  to  Improve  LLIS 


Category 

Dissemination 


Searchability 


More  Positive  Lessons 


Specific  Suggestion 

•  For  non-NASA  employees,  it  is  extremely  difficult  to  be  granted  access  on  a  project  basis.  Team 
members  on  my  project  must  apply  for  and  be  granted  access  on  an  individual  basis.  The  process  for 
access  is  tedious  to  say  the  least  and  this  discourages  routine  use  of  the  LLIS. 

•  Send  out  lessons  of  the  month. 

•  Include  a  bulletin  each  week  in  the  center  newsletters  about  a  lesson  learned. 

•  Send  out  alerts. 

•  Tailor  the  dissemination  or  notification  to  target  opportunities  for  benefit  rather  than  blanket  notification. 

Too  much  data  inhibits  information. _ 

•  Have  Mission  Assurance  Manager  or  Systems  Engineer  review  LLIS  on  periodic  basis  &  disseminate 
info  where  appropriate;  this  same  individual  could  review  fallure/discrepancy  reports  for  submission  to 
LLIS. 

•  Separate  safety  and  mishap  from  program,  technical,  etc,  lessons  learned.  Provide  possible  separate 
systems  for  each  category:  safety,  technical,  theoretical.  Commercial  product  usage,  contracting, 
project  management,  etc. 

•  Better  indexing  -  LLIS  search  usually  does  not  return  a  high  ratio  of  relevant  information 

•  Improved  search  engine. 

•  Categorize  the  lessons  into  categories  like  the  various  technical  disciplines  and  management 

disciplines,  so  that  it  would  be  easy  to  scan  the  ones  you  think  are  relevant  to  you. _ 

•  Need  more  “positive”  lessons  learned.  They  can  be  even  more  helpful  to  a  project  than  the  “negative” 
ones.  Even  if  a  project  avoids  “negatives”,  without  “positives”  they  may  not  follow  the  most  effective 
and  efficient  path  to  mission  success. 

•  When  NASA  has  a  failure,  there  are  all  sorts  of  reports  on  what  went  wrong  from  independent  review 
panels  (e.g.  Mars  Reports).  It  seems  there  would  definitely  be  some  benefit  from  similar  type  reports 
on  projects  that  were  successful,  pointing  out  what  these  projects  have  done  right. 

•  Instead  of  focusing  on  the  negative,  focus  on  the  positive  things.  There  are  some  really  well  run 
projects.  If  people  can  look  to  great  successes,  instead  of  worrying  about  failures,  I  believe  that  they 
will  try  to  emulate  those  successes.  NASA  has  always  been  known  as  a  risk  taker  because  of  what 
we  do.  We  have  no  one  else  to  look  to,  but  within  ourselves.  I’m  beginning  to  see  something  new  at 
NASA  -  fear  of  taking  a  risk.  As  long  as  risks  are  calculated,  and  the  project/program  buys-in  to 


In  discussions  with  NASA  officials,  we  found  there  was  general  agreement 
with  the  problems  and  suggested  improvements  for  lessons  learning  that 
were  identified  in  our  survey.  Officials  indicated  that  lessons  learning  has 
taken  on  greater  importance  in  recent  years  due  to  the  implementation  of 
more  programs  and  projects  under  the  FBC  strategy  and  the  continuing 
loss  of  agency  expertise  due  to  attrition.  They  acknowledge  that  LLIS  has 
not  been  an  effective  mechanism  for  agency-wide  sharing  of  lessons. 
Although  the  system  is  viewed  as  providing  a  useful  repository  for  storing 
lessons,  officials  agreed  with  managers’  concerns  about  the  difficulties 
involved  in  searching  the  system  and  finding  relevant  lessons,  the 
inconsistent  quality  of  information  contained  in  the  system,  and  the  lack  of 
lessons  about  positive  project  experiences.  However,  while  program  and 
project  managers’  suggested  improvements  would  help  increase  the 
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usability  of  LLIS,  they  do  not  target  some  of  the  more  fundamental 
problems  hampering  NASA’s  ability  to  share  lessons,  such  as  the  cultural 
barriers  that  persist. 
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The  limitations  in  NASA’s  ability  to  share  lessons  learned  point  toward 
two  underlying  problems:  cultural  resistance  to  sharing  knowledge  and  the 
lack  of  an  effective  strategic  framework  and  management  attention  for 
overcoming  such  resistance.  The  same  problems  often  hamper 
organizations  not  only  in  sharing  lessons  learned,  but  also  in  broader 
efforts  to  use  knowledge  as  a  tool  for  increasing  productivity  and  fostering 
innovation.  We  found  that  successful  industry  and  government 
organizations  have  learned  to  overcome  such  barriers  by  developing 
strategic  plans  that  provide  a  framework  for  how  knowledge  management 
will  be  implemented  and  by  securing  the  management  support  and 
commitment  needed  to  bring  these  plans  to  fruition.  Moreover,  they 
employ  incentives,  processes,  and  systems  designed  to  overcome 
resistance  and  other  cultural  impediments  to  knowledge  sharing.  Just  as 
employing  these  practices  enables  leading  organizations  to  maintain  a 
competitive  edge,  doing  so  can  better  position  NASA  to  perform  its  basic 
mission  of  exploring  space,  faster,  better,  and  cheaper. 


Principles  of 
Knowledge 
Management  Could 
Help  Mitigate 
Fundamental 
Weaknesses  in 
Lessons  Learning 


We  contacted  practitioners  of  knowledge  management  in  both  government 
and  industry  to  gain  insight  relative  to  how  the  application  of  knowledge 
management  principles  could  mitigate  the  fundamental  weakness  in 
lessons  learning  identified  by  our  survey.  This  effort  included  site  visits 
and  interviews  of  government  and  industry  knowledge  management 
practitioners,  analysis  of  published  case  studies,  and  review  of  available 
literature.  The  outcome  was  the  identification  of  common  characteristics 
inherent  in  the  design  and  implementation  of  knowledge  management 
programs  operated  and  maintained  by  government  and  commercial 
organizations.  Literature  searches  and  reviews  of  case  studies  also 
identified  general  principles  deemed  critical  to  the  success  of  knowledge 
management. 


Knowledge  Management  According  to  practitioners  of  knowledge  management,  the  first  and 
Must  Be  Linked  to  a  perhaps  most  important  element  of  knowledge  management  is  to  put  in 

Business  Plan  place  a  strategic  plan  that  makes  a  business  case  for  knowledge 

management.  This  involves  translating  the  abstract  concept  of  knowledge 
management  into  a  vision,  with  goals  and  a  roadmap  for  sharing  and  using 
knowledge  within  the  organization.  It  also  involves  developing  a  collective 
vision  that  is  long  term  and  a  commitment  by  senior  management  that  they 
will  see  the  plan  through. 


NASA  has  developed  a  strategic  plan  that  identifies  knowledge 
management  goals  and  objectives  the  agency  hopes  to  achieve.  NASA, 
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however,  has  not  developed  a  business  plan  for  achieving  implementation 
of  its  vision  and  goals.  Such  a  plan  at  a  minimum  should  address  the 
following: 

•  roles  and  responsibilities; 

•  knowledge  needs  and  how  they  it  relate  to  business  processes; 

•  the  role  of  information  technology; 

•  a  timetable  for  implementing  for  knowledge  management; 

•  resources  needed  for  implementing  a  knowledge  strategy 

•  cultural  barriers  to  learning; 

•  metrics  needed  for  tracking  and  measuring  results;  and 

•  training. 


Senior  Management 
Support  is  Key  to 
Knowledge 

Management’s  Success 


While  NASA  senior  management  has  shown  its  support  for  knowledge 
sharing  and  lessons  learning,  our  survey  nonetheless  indicates  that  many 
program  and  project  managers  believe  senior  management  support  is 
lacking.  Over  one-third  of  survey  respondents  indicated  that  the  effective 
utilization  of  lessons  learning  at  NASA  is  inhibited  because  senior 
management  lacks  commitment  to  its  use.  One  respondent  stated  that  if 
management  is  serious  about  capturing  and  using  lessons  learned,  then 
senior  management  must  be  seen  and  heard  implementing  the  policy. 

Of  the  organizations  we  observed,  most  had  senior  management  who  saw 
the  strategic  value  in  using  knowledge  and  communicated  to  their 
employees  that  sharing  knowledge  is  critical  to  their  success.^  At  Ford 
Motor  Company,  the  chief  executive  officer  serves  as  a  role  model  for 
knowledge  sharing,  personally  writing  weekly  e-mails  to  employees  with 
comments  on  his  thoughts  about  and  experiences  of  the  past  week.^ 
Moreover,  Ford  executives  provide  sponsorship  and  support  for 


^  Thomas  Davenport  and  Laurence  Prusak,  Working  Knowledge:  How  Organizations 
Manage  What  They  Know  (Boston:  Harvard  Business  School  Press, 1998). 

^  Ford  Motor  Company  1999  Annual  Report. 
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knowledge  management.  They  encourage  employee  participation  and 
recognize  employees  who  share  information. 

An  aerospace  company  believes  that  one  of  the  best  ways  to  disseminate 
lessons  learned  is  to  have  management  use  them.  This  means  that  for  the 
dissemination  of  lessons  learned  to  be  considered  important  by  all  the 
employees  in  the  company,  the  executive  team  must  understand  and  value 
them.  This  is  accomplished  by  including  as  part  of  its  executive  training 
course  items  such  as  teaching  managers  how  to  file  after  action  reports. 


Management  of  Knowledge  Although  NASA’s  policies  and  procedures  require  continuous  collection 
Sharing  Activities  sharing  of  lessons  learned,  NASA  has  not  established  a  central 

management  function  to  coordinate  and  direct  its  lessons  learning 
processes  and  systems.  Lessons  learning  activities  are  dispersed  across 
several  agency  components.  The  Office  of  Safety  and  Mission  Assurance 
has  designated  an  administrator  to  maintain  the  LLIS;  however, 
responsibility  for  LLIS  is  one  of  several  assigned  duties  for  the 
administrator.  As  a  result,  little  effort  is  devoted  to  managing  LLIS  beyond 
keeping  it  operating.  NASA  has  also  established  a  team  to  coordinate 
recent  knowledge  management  initiatives,  but  the  team  lacks  authority  to 
shape  the  direction  or  provide  resources  and  tools  to  improve  knowledge 
sharing. 

Our  research  indicates  that  organizations  that  have  implemented 
knowledge  sharing  programs  often  designate  a  knowledge  management 
steward.  For  example,  a  recent  article  on  knowledge  management  notes 
that  "...  a  job  title  at  Philip  Morris  is  knowledge  champion,  Monsanto  has 
a  director  of  knowledge  management,  and  Dow  Chemical  has  a  director  of 
intellectual  assets  management.”^  Once  such  a  position  has  been  created, 
the  steward  is  then  responsible  for  identifying  needs  and  opportunities  for 
improvement,  obtaining  and  distributing  resources,  applying  and  enforcing 
policies,  and  conducting  daily  operations.  At  Ford,  a  best  practices 
coordinator  manages  the  knowledge-sharing  program.  Similarly,  at  the 
World  Bank,  a  Knowledge  Management  Office  leads  an  organization-wide 
effort  at  knowledge  sharing. 


^  Verna  AUee,  Linking  People,  Learning  and  Performance  -12  F^nciples  of  Knowledge 
Management,  American  Society  for  Training  and  Development,  (2001). 
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Investing  in  Knowledge 
Sharing  Is  Viewed  as 
Important  by  Successful 
Organizations 


NASA  currently  invests  little  in  lessons  learning.  LLIS  costs  $50,000  to 
$100,000  annually  to  operate.  And,  according  to  NASA  officials,  funding  is 
insufficient  to  upgrade  LLIS  in  any  significant  way.  Although  there  is  no 
set  answer  as  to  the  appropriate  dollar  amount  that  should  be  devoted  to 
knowledge  management,  our  review  shows  that  those  organizations  that 
dedicate  resources  tend  to  be  more  successful  at  sharing  knowledge 
effectively.  Many  of  the  organizations  we  contacted  reported  significant 
investments  in  knowledge  management  activities.  For  example,  the  World 
Bank  invests  about  3  percent  of  its  total  administrative  budget  in 
knowledge  management.  These  resources  are  used  for  information 
technology  systems,  training,  and  dedicated  staff  to  facilitate  knowledge 
sharing. 


A  Corporate  Culture  That 
Encourages  Knowledge 
Sharing  Is  a  Key  Element 
for  Success 


Although  NASA  acknowledges  the  importance  of  having  an  organizational 
culture  that  promotes  teamwork  and  knowledge  sharing,  our  survey 
indicated  that  significant  cultural  barriers  within  the  agency  inhibit  an 
environment  and  culture  that  supports  continuous  learning.  Our  research 
indicates  that  practitioners  of  knowledge  management  foster  an 
environment  and  culture  that  support  continuous  learning.  Many 
knowledge  management  leaders  agree  that  knowledge  management  is  90 
percent  culture  and  10  percent  technology.'^  A  culture  of  knowledge 
sharing  can  be  encouraged  in  many  ways. 


Effective  Knowledge  Sharing  A  common  complaint  among  NASA  survey  respondents  is  the  lack  of  time 
Requires  Adequate  Time  sharing  lessons  learned.  Consequently,  knowledge-sharing  activities  at 

NASA  are  seen  as  simply  an  additional  burden  on  an  already  tight 
schedule.  Yet,  if  an  organization  values  knowledge  sharing  it  will  allow 
time  for  knowledge  sharing.  Knowledge  can  only  be  shared  when 
employees  are  given  adequate  time,  as  well  as  established  places  where 
they  can  actually  transfer  knowledge. 


Formal  And  Informal  Our  survey  indicated  that  much  of  the  knowledge  sharing  at  NASA  is  done 

Knowledge  Sharing  ou  an  informal  basis.  While  talking  informally  with  colleagues  is  one 

method  of  sharing  information,  it  is  not  necessarily  the  most  efficient  way. 
Such  informal  exchanges  do  not  allow  the  information  to  be  further 
disseminated  and  do  not  necessarily  allow  the  information  to  be  validated. 


Directorate  of  eBusiness  &  Knowledge  Management,  Office  of  the  Chief  Information 
Officer,  DOD  Knowledge  Management  Primer,  Department  of  Defense. 
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Information  exchanges,  however,  are  important  because  they  allow  people 
to  discuss  issues  spontaneously.  Many  of  the  organizations  that  we 
examined  instituted  more  formal  arrangements  for  sharing  information. 
Ford  and  the  World  Bank  have  established  “communities  of  practice”  as  a 
method  for  sharing  knowledge.  They  formed  groups  of  employees  who  are 
bound  together  by  shared  expertise  and  a  passion  for  joint  enterprise.^ 
These  communities  serve  as  knowledge  transfer  programs  because 
employees  can  share  information  regarding  a  given  issue.  At  the  World 
Bank  there  are  approximately  120  communities  of  practice — known  as 
“thematic  groups”  arranged  according  to  common  interests,  such  as 
“urban  poverty”  or  “urban  services.”  The  urban  services  group,  for 
example,  focuses  on  improving  the  living  conditions  in  slums  around  the 
world  by  capturing  and  sharing  past  experiences  and  adapting  them  to 
today’s  experiences. 

Incentives  for  Knowledge  Recently,  some  activities  at  NASA  centers  have  sought  to  find  ways  to 

Sharing  encourage  knowledge  sharing.  For  example,  scientists  at  NASA’s  Langley 

Research  Center  are  monetarily  rewarded  if  knowledge  they  capture  and 
share  is  re-used.  Nevertheless,  NASA  has  not  developed  a  plan  that  details 
the  principles,  requirements,  and  architecture  of  how  to  implement  its 
knowledge  sharing  strategy.  While  NASA  realizes  that  rewards  and 
recognition  should  be  part  of  its  initial  priority  area  for  knowledge 
management,  the  agency  has  not  yet  initiated  any  agency  wide  incentives 
for  knowledge  sharing. 

Most  of  the  organizations  that  we  examined  have  developed  incentives  for 
knowledge  sharing.  Ford  rewards  employees  who  submit  lessons  learned 
that  are  adopted  by  the  company.  Managers  are  also  encouraged  to  share 
because  they  are  evaluated  annually  on  the  basis  of  knowledge  sharing. 
Similarly,  the  World  Bank  has  made  learning  and  knowledge  sharing  part 
of  core  behaviors  covered  in  each  employee’s  performance  evaluation. 


Information  Is  Deemed  NASA  is  less  successful  at  infusing  lessons  learned  because  the  sharing  of 

Valuable  lessons  learned  is  not  highly  valued  by  some  program  and  project 

managers.  Successful  knowledge  sharing  organizations  share  knowledge 
because  they  view  it  as  critical  to  their  success.  At  Ford,  for  example, 
every  plant  is  responsible  for  making  a  5  percent  productivity  increase 
each  year.  Employees  refer  to  this  as  “task.”  Ford’s  knowledge-sharing 


^  Wenger,  Etienne  and  William  M.  Snyder.  “Communities  of  Practice:  The  Organizational 
Frontier.”  Harvard  Business  Review.  Jan/Feb.  2000:139-145. 
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database  (Best  Practice  Replication  system)  provides  managers  with 
suggestions  for  improving  efficiency,  which  in  turn  will  enable  managers 
to  meet  “task.”  Knowledge  is  shared  because  it  meets  a  critical  need. 
Moreover,  managers  at  Ford  use  the  lessons  provided  by  the  Best  Practice 
Replication  system  because  they  trust  the  information,  and  experts  within 
the  organization  have  checked  the  best  practice  in  the  database  for 
accuracy.®  This  is  key  to  the  Ford  program. 


Information  Technology  is 
Important,  but  Should  Not 
Be  the  Only  Mechanism  for 
Knowledge  Sharing 


A  pitfall  of  many  knowledge  management  programs  is  the  assumption  that 
a  database  will  automatically  lead  to  knowledge  sharing.  Indeed,  NASA’s 
agency-wide  effort  at  lessons  learning  is  in  practice  limited  to  LLIS.  At 
Ford,  its  electronic  database  is  supplemented  by  other  interactions.  Not 
only  do  employees  meet  at  community  of  practice  meetings,  but 
production  engineers  from  the  various  plants  also  meet  quarterly  to 
discuss  how  each  plant  has  implemented  best  practices.  The  World  Bank 
not  only  uses  a  variety  of  databases  to  share  information;  employees  also 
meet  face-to-face  at  thematic  group  meetings  and  at  knowledge  fairs. 


Well-designed  information  systems  and  databases,  however,  are  important 
to  facilitate  knowledge  sharing,  especially  for  organizations  such  as  NASA 
that  have  employees  located  at  multiple  centers  across  the  country. 
Organizations  with  effective  knowledge  management  have  designed  their 
database  systems  with  heavy  end-user  involvement.  The  individuals  who 
are  intended  to  use  the  databases  should  be  involved  in  the  development 
and  implementation  of  the  system  if  it  is  to  be  successful.  In  addition, 
organizations  often  use  dissemination  mechanisms  to  “push”  important 
and  relevant  information  to  the  right  people  rather  than  have  users  to 
search  for  the  information.  NASA’s  LLIS  requires  users  to  search  for 
possible  useful  lessons.  In  contrast,  at  the  Department  of  Energy  users 
receive  automatic  e-mail  alerts  regarding  important  lessons  learned. 
Similarly,  individuals  at  Ford  receive  new  best  practices  every  time  they 
open  the  database  system.  Also,  we  found  that  organizations  frequently 
dedicate  “gatekeepers”  to  manage  and  monitor  knowledge-sharing 
databases  in  order  to  keep  the  information  up  to  date  and  relevant.  At 
Ford,  a  “focal  point”  is  assigned  to  manage  the  best  practices  system  at 
each  plant.  The  focal  point  is  usually  a  production  engineer  appointed  by 
the  plant  manager.  They  receive  the  pushed  messages  and  also  enter  best 


®  Nancy  M.  Dixon,  Common  Knowledge:  How  Companies  Thrive  by  Sharing  What  They 
Know,  (Boston:  Harvard  Business  School  Press,  2000). 
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practices  derived  from  their  own  plant.  Similarly,  at  the  Department  of 
Energy  a  lessons  learned  coordinator  maintains  the  lessons  learned 
system  at  each  site. 
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The  failures  of  the  Mars  Polar  Lander  and  Climate  Orbiter  spacecraft 
raised  concern  that  NASA  was  not  learning  from  its  past  mistakes  and 
applying  lessons  learned  toward  future  mission  success.  Our  survey  of 
NASA  program  and  project  managers  and  subsequent  analysis  found  that 
there  are  weaknesses  in  NASA’s  processes,  procedures,  and  systems  for 
lessons  learning  and  knowledge  sharing.  Program  and  project  managers 
indicate  that  current  processes  and  systems  do  not  allow  them  to  retrieve 
the  right  lesson  at  the  right  time.  Managers  also  report  that  they  are  not 
very  knowledgeable  about  lessons  generated  from  other  centers  and 
programs.  This  is  partly  due  to  the  fact  that  NASA’s  LLIS  is  not  widely 
used.  As  a  result,  there  is  no  assurance  that  lessons  learned  from  past 
mishaps  or  program  successes  are  being  applied  to  current  programs  and 
projects. 

NASA  has  recognized  that  it  must  improve  its  lessons  learning  and 
knowledge  sharing  capabilities.  The  mandate  to  deliver  leaner,  more 
demanding,  and  increasingly  more  complex  missions  has  resulted  in  a 
greater  dependency  on  creating  multi-disciplinary  teams,  building 
alliances  with  contractors,  and  being  able  to  quickly  link  to  and  learn  from 
other  agency  activities.  Following  a  series  of  mission  failures  and  mishaps, 
the  NASA  Integrated  Action  Team  reported  that  the  continuous  capture 
and  application  of  project  knowledge  and  lessons  learned  must  become  a 
core  business  process  within  the  agency’s  program  and  project 
management  environment.  More  importantly,  the  team  stated  that  program 
and  project  managers  must  regularly  use  knowledge  management  tools  to 
apply  previous  lessons  learned  to  their  projects. 

NASA  has  taken  steps  to  promote  the  capture  and  sharing  of  lessons  and 
other  knowledge  within  the  agency  through  knowledge  management 
practices.  According  to  NASA  officials,  knowledge  management  has  the 
potential  to  rectify  many  of  the  weaknesses  identified  by  our  survey 
through  more  effective  management  of  the  agency’s  knowledge  resources. 
Current  knowledge  management  initiatives  by  NASA  should  lead  to 
improvements  in  knowledge  sharing,  but  the  lack  of  a  coordinated  and 
well-supported  effort  will  limit  knowledge  sharing  agency-wide.  Stronger 
commitment  and  efforts  are  needed  to  create  an  environment  that 
encourages  knowledge  sharing.  To  achieve  this  environment,  NASA  must 
overcome  communication  barriers  and  create  opportunities  where  open 
and  candid  communications  are  the  norm  and  knowledge  sharing  is 
valued.  In  addition,  time  and  resources  must  be  dedicated  in  order  for 
lessons  learning  and  knowledge  sharing  to  occur.  Furthermore, 
improvements  to  information  technology  systems  such  as  LLIS  are  needed 
to  facilitate  the  collection,  dissemination,  and  application  of  knowledge. 
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Recommendations  for 
Executive  Action 

NASA  needs  to  strengthen  its  lessons  learning  in  the  context  of  its  overall 
efforts  to  develop  and  implement  an  effective  knowledge  management 
program.  Improvement  of  NASA’s  lessons  learning  processes  and  systems 
can  help  to  ensure  that  knowledge  is  gained  from  past  experiences  and 
applied  to  future  missions. 

We  recommend  that  the  NASA  administrator  strengthen  the  agency’s 
lessons  learning  processes  and  systems  by  taking  actions  in  the  following 
areas: 

Strategic  Planning 

•  Articulate  the  relationship  between  lessons  learning  and  knowledge 
management  through  development  of  an  implementation  plan  for 
knowledge  management. 

Coordination 

•  Designate  a  lesson  learned  manager  to  lead  and  coordinate  all  agency 
lessons  learning  efforts. 

•  Establish  functional  and  technical  linkages  among  the  various  center- 
level  and  program-level  lessons  learning  systems. 

Culture 

•  Develop  ways  to  broaden  and  implement  mentoring  and  “storytelling” 
as  additional  mechanisms  for  lessons  learning. 

•  Identify  incentives  to  encourage  more  collection  and  sharing  of  lessons 
among  employees  and  teams,  such  as  links  to  performance  evaluations 
and  awards. 

LLIS  Enhancement 

Enhance  LLIS  by 

•  coding  information  and  developing  an  easier  search  capability  to  allow 
users  to  identify  relevant  lessons; 

•  including  more  positive  lessons; 

•  providing  a  means  to  disseminate  key  lessons  to  users;  and 

•  soliciting  user  input  on  an  ongoing  basis. 
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Performance  •  Track  and  report  on  the  effectiveness  of  the  agency’s  lessons  learning 

Measurement  efforts  using  objective  performance  metrics. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  NASA  generally  concurred 
with  our  recommendations  for  improving  the  agency’s  lessons  learned 
processes  and  systems.  NASA  stated  that  it  must  do  a  better  job  of 
communicating  the  various  lessons  learned  sources  to  employees, 
improving  mechanisms  to  link  these  sources,  and  ensuring  appropriate 
training  for  employees  in  order  to  maximize  lessons  learning.  NASA 
further  indicated  that  it  will  develop  plans  to  implement  our  report 
recommendations.  NASA’s  comments  are  reprinted  in  appendix  1. 
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Nalional  Aeronajtics  ard 
Space  Administration 
Office  (yfihe  Administrator 
Washington.  DC  20S46-0001 


Mr.  Allen  Li 
Director 

Acquisition  and  Sourcing  Management 
U.  S.  General  Accounting  Office 
Washington,  DC  20648 


JAN  I  4  2002 


Dear  Mr,  Li: 

We  appreciate  the  opportunity  to  review  and  provide  comments  to  the  draft 
report,  entitled,  NASA:  Better  Mechanisms  Needed  far  Sharing  Lessons  Learned. 
We  have  reviewed  the  report,  and  although  may  not  agree  with  the  cordusions  that 
correlate  Mars  Mission  failures  with  deficiencies  in  the  application  of  lessons 
learned,  we  are  essentially  in  agreement  with  the  recommendabons  for  improving 
our  Lessons  Learned  Process.  These  recommendations  are  generally  on  the  mark, 
and  their  intent  will  be  implemented.  There  are  several  initiabves  currently 
underway,  and  implementation  plans  are  being  developed. 

We  continue  to  evaluate  NASA  processes  and  will  do  the  same  for  lessons 
learned.  Major  revisions  to  NASA  Procedures  and  Guidelines  (NPG)  7120,5,  NASA 
Program  and  Project  Management  Processes  and  Requirements,  are  in  the  final 
stages  and  Include  more  clarity  In  the  use  and  scope  of  the  Lessons  Learned 
Information  System  (LLIS).  Since  NASA  uses  numerous  sources  for  lessons 
learned,  LLIS  will  no  longer  be  referred  to  as  the  primary  source,  but  one  of  many. 

Programs  and  projects  benefit  from  lessons  learned  day  to  day  in  many 
different  forms.  Some  of  these  include  status  reporting,  programmatic  and  technical 
reviews,  and  functional  staff  meetings.  The  more  fomnal  and  widespread  forms 
indude  technical  standards  and  updating  NPG’s  and  other  documentation.  The 
various  NASA  training  programs  also  incorporate  lessons  learned  into  their 
cumicula,  including  “storytelling  '  by  current  and  retired  NASA  program  and  project 
managers.  One  forum  Is  designed  specifically  for  sharing  lessons  learned  and  is 
titled  Project  Management  Shared  Experiences  Program.  The  overall  objective  of 
this  program  is  to  provide  a  fonjm  to  understand  key  initiatives  irTfluencing  NASA 
project  management  and  for  project  people  to  share  knowledge,  experiences,  and 
creative  approaches  to  project  management. 

NASA  must  do  a  better  job  of  communicating  the  various  lessons  learned 
sources,  improving  mechanisms  by  which  these  various  sources  are  linked,  and 
encouraging  the  [croper  training  for  employees  in  order  to  maximize  the  ability  of 
current  activities  to  draw  upon  the  lessons  of  others.  We  plan  continued 
improvement  in  these  areas. 
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We  appreciate  the  GAO  assessment  effort  and  subsequent 
recommendations  on  improving  NASA's  iessons  iearned  system.  This  is  a  good 
example  of  agencies  teaming  for  improvement. 

At  this  time,  we  provide  the  following  response,  if  the  recommendations 
contained  in  the  draft  report  are  retained  in  the  final  report. 

GAO  Recommendation  1 :  Articulate  the  relationship  between  lessons  learning 
and  knowledge  management  through  an  implementation  plan  for  knowledge 
management. 

NASA  Response:  Concur.  NASA  has  developed  a  "Strategic  Plan  for 
Knowledge  Management"  and  "Near-Term  Implementation  Opportunities  for 
Knowledge  Management"  which  articulates  the  relationship  between  lessons 
learning  and  knowledge  management.  The  Near-Term  Implementation 
Opportunities  plan  will  be  revised  to  strengthen  the  linkage  between  knowledge 
management  activities  and  processes  for  lessons  learning.  Some  of  the 
implementation  activities  are  underway  and  others  will  be  initiated  as  the  overall 
implementation  progresses. 

GAO  Recommendation  2:  Designate  a  lessons  learned  manager  to  lead  and 
coordinate  all  agency  lessons  learning  efforts. 

NASA  Response:  Concur.  NASA  is  planning  on  designating  a  lessons  learned 
manager. 

GAO  Recommendation  3:  Establish  functional  and  technical  linkages  among  the 
various  center-level  and  program-level  lessons  learning  systems. 

NASA  Response:  Concur.  As  NASA  progresses  in  the  implementation  of 
knowledge  management,  methods  to  link  the  multiple  lessons  learning  systems  will 
be  pursued  and  incorporated. 

GAO  Recommendation  4:  Develop  ways  to  broaden  and  implement  mentoring 
and  "storytelling”  as  additional  mechanisms  for  lessons  learning. 

NASA  Response:  Concur.  NASA  will  continue  to  hold  several  project 
management  knowledge  sharing  forums,  the  annual  publication  of  several  issues  of 
the  online  Academy  Sharing  Knowledge  Journal,  and  the  development  of  additional 
case  studies  and  online  knowledge  sharing  resources.  Efforts  are  also  unden/vay  to 
improve  the  systematic  identification  and  utilization  of  NASA  leaders  as  mentors 
and  teachers,  and  processes  and  procedures  will  be  completed  in  2002. 
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GAO  Recommendation  5:  Identify  incentives  to  encourage  more  coiiection  and 
sharing  of  lessons  among  empioyees  and  teams,  such  as  iinks  to  performance 
evaiuations  and  awards. 

NASA  Response:  Concur,  in  October  2001 ,  the  Agency  began  a  new  study-- 
"Aiigning  Recognition  and  Awards  with  the  Agency's  Vaiues  and  Expectations."  A 
significant  issue  of  this  study  scope  is  "What  recognition  practices  in  a  technology- 
based  organizationai  setting  wouid  drive  and  support  particuiar  initiatives  in 
teaming,  knowledge  sharing,  and  organizational  learning."  Recommendations 
resulting  from  this  study  will  be  proposed  by  the  end  of  this  fiscal  year. 

GAO  Recommendation  6:  Enhance  LLIS  by  coding  information  and  developing  an 
easier  search  capability  to  allow  users  to  identify  relevant  lessons,  including  more 
positive  lessons,  providing  a  means  to  disseminate  key  lessons  to  users;  and 
soliciting  user  input  on  an  ongoing  basis. 

NASA  Response:  Concur.  Since  the  inception  of  the  LLIS  in  1995  we  have 
offered  users  an  extremely  easy-to-use  “keyword  ’  search  capability  for  all  data  in 
the  LLIS  that  functions  in  the  same  way  as  an  Internet  search  browser  such  as 
“Yahoo”  or  “Lycos”.  Since  1999,  we  have  also  offered  our  users  an  advanced 
search  capability.  Currently,  this  advanced  search  capability  allows  users  to  search 
on  several  combinations  of  criteria  (NASA  Center/Facility,  Enterprise,  and 
Program/Project  life-cycle  stage),  including  keyword  search.  This  collection  of 
search  combinations  gives  our  users  unparalleled  searching  flexibility. 

Recent  improvements  now  provide  the  capacity  to  disseminate  key  lessons  to  users 
with  our  new  “push”  capability.  NASA  will  continue  to  study  improvement 
possibilities  for  the  LLIS  to  ease  the  system’s  use. 

The  LLIS  web  site  offers  several  means  by  which  users  can  presently  contact  us  to 
provide  comments  or  feedback  on  the  system.  We  have  a  “submit  a  suggestion 
about  the  LLIS”  link  and  a  “comment  on  a  lesson”  link.  We  also  have  a  hotline/help 
desk  telephone  number:  301-286-LLIS. 

GAO  Recommendation  7:  Track  and  report  on  the  effectiveness  of  the  agency’s 
lessons  learning  efforts  using  objective  performance  metrics. 

NASA  Response:  Concur.  Although  challenging,  NASA  is  pursuing  the 
appropriate  metrics  to  measure  the  effectiveness  of  our  lessons  learning  efforts. 

We  expect  to  have  this  in  place  later  this  year. 
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Thank  you  again  for  the  opportunity  to  review  this  draft  report.  Should  you 
have  any  questions,  please  call  Keith  Hudkins,  Deputy  Chief  Engineer,  at  (202) 
358-1 823.  The  point  of  contact  for  this  audit  in  the  Office  of  the  Chief  Engineer  is 
Gregory  Robinson;  he  can  be  reached  at  (202)  358-2541 . 

Sincerely, 

Michael  D.  Christensen 

Associate  Deputy  Administrator  for  Institutions 

cc: 

AE/Mr.  Keegan 
AE/Mr.  Hudkins 
AE/Mr.  Robinson 
AO/Mr.  Holcomb 
AO/Ms.  Hibbert 
F/Ms.  Novak 
FT/Ms.  Acoveno 
FP/Ms.  Peterson 
G/Mr.  Stephens  (Acting) 

JMr.  Sutton 
JM/Ms.  Flickinger 
JM/Mr.  Werner 
L/Mr.  Bingham  (Acting) 

M/Mr.  Gregory  (Acting) 

Q/Dr.  Greenfield  (Acting) 

R/Mr.  Venneri 
U/Dr.  Olsen  (Acting) 

Y/Dr.  Asrar 
ARC/Dr.  McDonald 
DFRC/Mr.  Petersen 
GRC/Mr.  Campbell 
GSFC/Mr.  Diaz 
JSC/Mr.  Estess  (Acting) 

KSC/Mr.  Bridges 
LaRC/Dr.  Creedon 
MSFC/Mr.  Stephenson 
SSC/Mr.  Craig  (Acting) 

JPL/Dr.  Elachi 
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GAO’S  Mission 

■ 

The  General  Accounting  Office,  the  investigative  arm  of  Congress,  exists  to 
support  Congress  in  meeting  its  constitutional  responsibilities  and  to  help 
improve  the  performance  and  accountability  of  the  federal  government  for  the 
American  people.  GAO  examines  the  use  of  public  funds;  evaluates  federal 
programs  and  policies;  and  provides  analyses,  recommendations,  and  other 
assistance  to  help  Congress  make  informed  oversight,  policy,  and  funding 
decisions.  GAO’s  commitment  to  good  government  is  reflected  in  its  core  values 
of  accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

■ - 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  is  through  the 
Internet.  GAO’s  Web  site  (www.gao.gov)  contains  abstracts  and  fuU-text  files  of 
current  reports  and  testimony  and  an  expanding  archive  of  older  products.  The 

Web  site  features  a  search  engine  to  help  you  locate  documents  using  key  words 
and  phrases.  You  can  print  these  documents  in  their  entirety,  including  charts  and 
other  graphics. 

Each  day,  GAO  issues  a  list  of  newly  released  reports,  testimony,  and 
correspondence.  GAO  posts  this  list,  known  as  “Today’s  Reports,”  on  its  Web  site 
daily.  The  list  contains  links  to  the  fuU-text  document  files.  To  have  GAO  e-mail 
this  list  to  you  every  afternoon,  go  to  www.gao.gov  and  select  "Subscribe  to  daily 
e-mail  alert  for  newly  released  products"  under  the  GAO  Reports  heading. 

Order  by  Mail  or  Phone 

The  first  copy  of  each  printed  report  is  free.  Additional  copies  are  $2  each.  A 
check  or  money  order  should  be  made  out  to  the  Superintendent  of  Documents. 
GAO  also  accepts  VISA  and  Mastercard.  Orders  for  100  or  more  copies  mailed  to  a 
single  address  are  discounted  25  percent.  Orders  should  be  sent  to: 

U.S.  General  Accounting  Office 

P.O.  Box  37050 

Washington,  D.C.  20013 

To  order  by  phone:  Voice:  (202)  512-6000 

TDD:  (202)  512-2537 

Fax:  (202)  512-6061 

Visit  GAO’s  Document 
Distribution  Center 

GAO  Building 

Room  1100,  700  4th  Street,  NW  (comer  of  4th  and  G  Streets,  NW) 

Washington,  D.C.  20013 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

■ 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm. 

E-mail:  fraudnet@gao.gov,  or 

l-(800)  424-5454  or  (202)  512-7470  (automated  answering  system). 

Public  Affairs 

■ - 

Jeff  Nelligan,  Managing  Director,  NelliganJ@gao.gov  (202)  512-4800 

U.S.  General  Accounting  Office,  441  G.  Street  NW,  Room  7149, 

Washington,  D.C.  20548 
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